AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

m365-agent-evaluator

Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts", "add multi-turn tests", "tune evaluator thresholds", "why is my agent failing", or "set up eval environment variables".

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--m365-agent-evaluator at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 15

references/output-schema.md

Output schema reference

The current runevals --output <file> JSON output is schema-compatible with the eval document format. It is not the older { "summary": ..., "results": [...] } shape.

Use references\output-schema.json for a compact validation-oriented schema and references\prompts-schema.json for the full package schema.

JSON output

Typical JSON output:

{
  "schemaVersion": "1.2.0",
  "metadata": {
    "evaluatedAt": "2025-01-01T00:00:00Z",
    "agentId": "00000000-0000-0000-0000-000000000000",
    "agentName": "Contoso Agent",
    "cliVersion": "1.5.0-preview.1"
  },
  "default_evaluators": {},
  "items": [
    {
      "prompt": "What can this agent help me with?",
      "response": "The agent response.",
      "expected_response": "The expected behavior.",
      "scores": {
        "relevance": {
          "score": 4,
          "result": "pass",
          "threshold": 3
        },
        "coherence": {
          "score": 5,
          "result": "pass",
          "threshold": 3
        }
      }
    }
  ]
}

Multi-turn output uses an item with turns and may include summary.

Score object

Scores are sparse. Keys appear only for evaluators that ran.

| Score key | Value shape | |---|---| | relevance | { "score": 1-5, "result": "pass" | "fail", "threshold": number } | | coherence | { "score": 1-5, "result": "pass" | "fail", "threshold": number } | | groundedness | { "score": 1-5, "result": "pass" | "fail", "threshold": number } | | similarity | { "score": 1-5, "result": "pass" | "fail", "threshold": number } | | citations | { "count": number, "result": "pass" | "fail", "threshold": number } | | exactMatch | { "score": 0 | 1, "result": "pass" | "fail", "threshold": number } | | partialMatch | { "score": 0.0-1.0, "result": "pass" | "fail", "threshold": number } |

Do not treat missing score keys as failures.

CSV output

CSV reports include:

  • metadata comments at the top,
  • aggregate statistics,
  • single-turn sections,
  • multi-turn sections,
  • serialized score JSON.

Use CSV when the user wants spreadsheet review or lightweight automation without parsing full JSON.

HTML output

HTML reports are for human review and may open in the default browser. Treat them as sensitive because they can contain prompts, retrieved data, and agent responses.

Automation guidance

When parsing JSON results:

  1. Read items.
  2. For each item, handle either single-turn prompt or multi-turn turns.
  3. Check only scores keys that exist.
  4. Treat setup/auth/model/schema errors separately from quality scores.
  5. Compare runs only when datasets, evaluators, thresholds, and model configuration are stable.