AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

m365-agent-evaluator

Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts", "add multi-turn tests", "tune evaluator thresholds", "why is my agent failing", or "set up eval environment variables".

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--m365-agent-evaluator at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 15

references/guardrails.md

Guardrails and troubleshooting

Use this reference before writing files, handling secrets, running state-changing commands, or diagnosing failures.

Safety rules

  1. Never print, commit, or summarize secret values from .env.local.user, env\.env.local.user, env\.env.<environment>, system environment variables, or terminal output.
  2. Treat prompts, agent responses, retrieved grounding data, HTML reports, CSV files, JSON results, and debug logs as potentially sensitive.
  3. Ask before overwriting an existing dataset or report.
  4. Prefer source datasets under evals\ and local run outputs under .evals\.
  5. Keep generated run outputs out of commits unless the user explicitly confirms they are sanitized and should be committed.
  6. Do not run real tenant-dependent evaluations without user-approved tenant, deployed-agent, and Azure OpenAI configuration.

File writes

Safe defaults:

| File | Default behavior | |---|---| | evals\evals.json | Ask before overwrite; this may be committed as a regression suite. | | .evals\*.json | Generated result output; usually local-only. | | .evals\*.csv | Generated result output; usually local-only. | | .evals\*.html | Generated result output; usually local-only and may contain response content. | | .env.local | Non-secret ATK config only. | | .env.local.user, env\.env.local.user, env\.env.<environment> | Secrets/local config; never commit or display values. |

When generating a dataset, write to a new file or a temporary file first, then rename after validation.

Commands that change local state

Warn the user before running:

npx -y --package @microsoft/m365-copilot-eval@latest runevals --cache-clear
npx -y --package @microsoft/m365-copilot-eval@latest runevals --signout

--cache-clear removes local cache data. --signout clears the local auth session and may require the user to sign in again.

Setup troubleshooting

| Symptom | Likely cause | Action | |---|---|---| | EULA prompt or refusal | EULA not accepted | Run npx -y --package @microsoft/m365-copilot-eval@latest runevals accept-eula. | | This version of the M365 Evals CLI has stopped working and must be updated | The shell resolved an outdated bare/global runevals shim | Use npx -y --package @microsoft/m365-copilot-eval@latest runevals --version; ask before uninstalling global packages or deleting shims. | | node or package startup error | Node version too old | Upgrade to Node.js 24.12.0 or newer. | | Agent cannot be resolved | Missing or wrong M365_TITLE_ID, M365_AGENT_ID, --m365-agent-id, or --env | Pass --m365-agent-id explicitly and verify the selected env file. | | Authentication failure | Not signed in, unsupported OS auth path, missing M365 Copilot license, or tenant admin consent missing | Validate signed-in account, license, and consent. | | Azure evaluator failure | Missing endpoint/key/model/API version | Check env keys exist without printing values; prefer gpt-4o-mini. | | Schema validation error | Old dataset format or unsupported evaluator | Use root items and public evaluator names only. | | Network/proxy errors | Enterprise network blocks auth/model calls | Ask the user to validate network/proxy access; do not retry with exposed secrets. |

Quality troubleshooting

Do not treat every failed run as an agent bug.

| Failure type | Category | |---|---| | Missing env, auth, consent, EULA, schema, or network | Setup failure | | Low relevance, coherence, groundedness, similarity, citations, exact match, or partial match | Agent/eval quality signal | | Ambiguous prompt or overly strict expected response | Eval dataset issue | | Missing source data or inaccessible connector | Capability/data issue |

Fix setup failures first. Then analyze quality signals with references\result-analysis.md.

Debug logging

Use debug logging only when needed:

npx -y --package @microsoft/m365-copilot-eval@latest runevals --log-level debug --prompts-file evals\evals.json --output .evals\debug.json

Debug output may expose prompts, responses, URLs, identifiers, or retrieved content. Do not paste logs into chat or commit them unless the user confirms they are safe.