AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

m365-agent-evaluator

Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts", "add multi-turn tests", "tune evaluator thresholds", "why is my agent failing", or "set up eval environment variables".

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--m365-agent-evaluator at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 15

examples/basic-generation.md

Example: create a starter eval dataset

User intent: "Create eval prompts for my M365 Copilot agent."

Steps

  1. Confirm the target agent scenario and whether the repo already has evals\evals.json.
  2. Load references\eval-templates.md and references\pra-framework.md.
  3. Create a schema 1.2.0 dataset with root items.
  4. Save it under evals\evals.json unless the user asks for another path.

Starter file

{
  "schemaVersion": "1.2.0",
  "metadata": {
    "name": "Starter M365 Copilot agent evals",
    "description": "Core smoke tests for agent scope, grounding, and response quality.",
    "tags": ["starter", "regression"]
  },
  "default_evaluators": {
    "Relevance": {},
    "Coherence": {}
  },
  "items": [
    {
      "prompt": "What can this agent help me with?",
      "expected_response": "The agent explains its supported scope without claiming unsupported capabilities."
    },
    {
      "prompt": "Summarize the latest status for the project using available sources.",
      "expected_response": "The agent summarizes available status, distinguishes known facts from missing data, and avoids unsupported claims.",
      "evaluators": {
        "Groundedness": {
          "threshold": 3
        }
      },
      "evaluators_mode": "extend"
    },
    {
      "prompt": "List the open action items with owners.",
      "expected_response": "The agent lists action items and owners only when source data supports them.",
      "evaluators": {
        "Citations": {
          "threshold": 1
        }
      },
      "evaluators_mode": "extend"
    }
  ]
}

First safe command

npx -y --package @microsoft/m365-copilot-eval@latest runevals --init-only

Run real evals only after the user confirms tenant, agent, and Azure OpenAI configuration is ready.