AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

m365-agent-evaluator

Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts", "add multi-turn tests", "tune evaluator thresholds", "why is my agent failing", or "set up eval environment variables".

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--m365-agent-evaluator at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 15

examples/analyze-failures.md

Example: analyze existing failures

User intent: "Here is my eval output. What should I fix?"

Inputs

Expected files:

.evals\latest.json
evals\evals.json

Do not paste raw prompts, responses, retrieved data, or logs into chat if they may contain sensitive content.

Process

  1. Load references\result-analysis.md and references\remediation-patterns.md.
  2. Inspect metadata for CLI version, agent ID/name, and evaluated time.
  3. Inspect each item under items.
  4. For each item, read sparse scores keys such as relevance, coherence, groundedness, similarity, citations, exactMatch, and partialMatch.
  5. Treat missing score keys as "not configured", not failed.
  6. Summarize the smallest targeted changes.

Example finding

Primary issue: Citation behavior is inconsistent.
Evidence: The citation evaluator failed on prompts that ask for workplace-source summaries, while relevance and coherence passed.
Recommended change: Add an instruction requiring citations for source-backed summaries and verify the agent path can surface citations.
Expected effect: `citations` should meet the minimum count threshold without changing the response content.

Common false positives

  • Expected response is too specific for a legitimately variable answer.
  • ExactMatch is used for natural language text.
  • A prompt assumes source data that is not available to the deployed agent.
  • The run failed during auth, schema validation, or evaluator-model setup.