AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

m365-agent-evaluator

Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts", "add multi-turn tests", "tune evaluator thresholds", "why is my agent failing", or "set up eval environment variables".

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--m365-agent-evaluator at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 15

examples/missing-instructions.md

Example: missing or weak instructions

User intent: "My agent gives vague answers in evals."

Symptoms

  • relevance passes but coherence fails.
  • groundedness fails because the agent invents missing facts.
  • similarity fails because the answer omits required structure or decisions.
  • Follow-up turns fail after a successful first turn.

Diagnosis

Check whether the agent instructions specify:

  1. supported scenarios and boundaries,
  2. source-grounding expectations,
  3. citation expectations,
  4. response format,
  5. behavior when data is missing,
  6. follow-up context handling.

Suggested instruction additions

Use only available workplace sources when answering source-backed questions. If the sources do not contain enough evidence, say what is missing instead of guessing.
For project-status answers, use this structure: Summary, Evidence, Risks, Next actions. Keep the answer concise and include citations when available.
For follow-up questions, preserve the project, customer, and time window from the prior turn unless the user changes them.

Matching eval update

Add or keep regression prompts that test the new instruction:

{
  "prompt": "Summarize the latest status for the project and include citations.",
  "expected_response": "The agent summarizes only source-backed status, cites available sources, and states when evidence is missing.",
  "evaluators": {
    "Groundedness": {
      "threshold": 4
    },
    "Citations": {
      "threshold": 1
    }
  },
  "evaluators_mode": "extend"
}