AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

m365-agent-evaluator

Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts", "add multi-turn tests", "tune evaluator thresholds", "why is my agent failing", or "set up eval environment variables".

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--m365-agent-evaluator at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 15

references/eval-templates.md

Evaluation dataset templates

Use these templates when creating or editing @microsoft/m365-copilot-eval datasets. The current public schema uses schemaVersion: "1.2.0" and a root items array.

Use references\prompts-schema.json as the local schema source.

Minimal single-turn dataset

{
  "schemaVersion": "1.2.0",
  "metadata": {
    "name": "Agent starter evaluation",
    "description": "Smoke tests for core agent behavior.",
    "tags": ["starter", "single-turn"]
  },
  "default_evaluators": {
    "Relevance": {},
    "Coherence": {}
  },
  "items": [
    {
      "prompt": "What can this agent help me with?",
      "expected_response": "The agent explains its supported scope without claiming unsupported capabilities."
    },
    {
      "prompt": "Summarize the latest status for Contoso renewal.",
      "expected_response": "The agent summarizes available status and avoids inventing facts when source data is unavailable."
    }
  ]
}

Single-turn item with context and groundedness

Use context when you have source material the answer must stay grounded in.

{
  "prompt": "What is the renewal deadline?",
  "expected_response": "The renewal deadline is May 31.",
  "context": "The Contoso renewal brief says the renewal deadline is May 31.",
  "evaluators": {
    "Groundedness": {
      "threshold": 4
    },
    "Similarity": {
      "threshold": 3
    }
  },
  "evaluators_mode": "extend"
}

evaluators_mode: "extend" adds item-level evaluators to default_evaluators. Use "replace" when the item should use only the item-level evaluator set.

Evaluator threshold examples

Public configurable evaluator names are case-sensitive:

{
  "default_evaluators": {
    "Relevance": {
      "threshold": 3
    },
    "Coherence": {
      "threshold": 3
    },
    "Groundedness": {
      "threshold": 3
    },
    "Similarity": {
      "threshold": 3
    },
    "Citations": {
      "threshold": 1
    },
    "PartialMatch": {
      "threshold": 0.5
    }
  }
}

Use ExactMatch sparingly. It is best for deterministic values such as IDs, dates, or fixed policy labels:

{
  "prompt": "Return only the ticket ID for the escalation.",
  "expected_response": "INC-12345",
  "evaluators": {
    "ExactMatch": {}
  },
  "evaluators_mode": "replace"
}

Multi-turn dataset

Use multi-turn items when the agent must preserve context across a conversation. The schema allows up to 20 turns.

{
  "schemaVersion": "1.2.0",
  "metadata": {
    "name": "Multi-turn follow-up suite",
    "tags": ["multi-turn", "regression"]
  },
  "default_evaluators": {
    "Relevance": {},
    "Coherence": {}
  },
  "items": [
    {
      "name": "Follow-up retains project context",
      "description": "The agent should remember that the user is discussing the Contoso renewal.",
      "conversation_id": "contoso-renewal-followup",
      "turns": [
        {
          "prompt": "What is the latest status for the Contoso renewal?",
          "expected_response": "The agent gives the available Contoso renewal status without inventing missing details."
        },
        {
          "prompt": "Who owns the next step?",
          "expected_response": "The agent answers in the context of the Contoso renewal and cites or qualifies the source of the owner."
        }
      ]
    }
  ]
}

Do not put top-level prompt on a multi-turn item. Put prompts inside turns.

PRA scenario design

Use PRA to choose what to test, then express tests with public evaluators.

| PRA area | Test intent | Useful evaluators | |---|---|---| | Perceive | Finds the right source, respects available context, uses citations where required | Groundedness, Citations, Relevance | | Reason | Synthesizes, follows instructions, handles ambiguity, avoids hallucination | Relevance, Coherence, Similarity | | Act | Performs or describes declared capabilities accurately | Relevance, Coherence, ExactMatch, PartialMatch, Similarity |

Do not add legacy/private evaluator names to generated datasets unless a current authoritative public schema includes them.

Tags and custom metadata

Use root metadata.tags for suite-level tags. Use extensions for local metadata that the CLI should preserve:

{
  "prompt": "List the open actions from the last planning discussion.",
  "expected_response": "The agent lists action items with owners when available.",
  "extensions": {
    "scenario": "action-items",
    "risk": "hallucinated-owner",
    "priority": "high"
  }
}

Authoring checklist

  1. Use schemaVersion: "1.2.0" and root items.
  2. Include clear expected_response text for every item where comparison matters.
  3. Keep prompts realistic but sanitized.
  4. Use only public evaluator names: Relevance, Coherence, Groundedness, Similarity, Citations, ExactMatch, PartialMatch.
  5. Use Citations as a minimum citation count, not as a 1-5 LLM score.
  6. Prefer Similarity or PartialMatch for flexible expected answers; use ExactMatch only when exact text is intended.
  7. Keep generated datasets in evals\; write run outputs under .evals\.