AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

m365-agent-evaluator

Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts", "add multi-turn tests", "tune evaluator thresholds", "why is my agent failing", or "set up eval environment variables".

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--m365-agent-evaluator at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 15

examples/run-and-analyze.md

Example: run evals and analyze results

User intent: "Run my evals and tell me why the agent is failing."

Safe preflight

node --version
npx -y --package @microsoft/m365-copilot-eval@latest runevals --version
npx -y --package @microsoft/m365-copilot-eval@latest runevals --help

Confirm env files exist without printing values. For first-time setup:

npx -y --package @microsoft/m365-copilot-eval@latest runevals accept-eula
npx -y --package @microsoft/m365-copilot-eval@latest runevals --init-only

Run with JSON output

npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --concurrency 1 --output .evals\latest.json

Use --concurrency 1 for debugging. Increase up to 5 only after setup is stable.

Optional human report

npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --output .evals\latest.html

Analysis approach

  1. Load references\result-analysis.md.
  2. Parse items from the JSON output.
  3. Check only score keys that exist.
  4. Separate setup/auth/model/schema failures from quality failures.
  5. Group quality failures by likely fix: instructions, grounding, citations, expected response, or capability gap.

Example response:

The main issue is grounding: two prompts passed relevance/coherence but failed groundedness. The agent answered with plausible project facts that were not present in the provided sources. Recommended change: add an instruction to answer only from retrieved workplace sources and say what is missing when evidence is insufficient.