AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

microsoft-foundry

Build, deploy, evaluate, optimize, fine-tune, and manage Microsoft Foundry agents, models, and resources end to end. USE FOR: foundry, azd ai agent, azd provision/deploy, hosted agent scaffold/develop/run/deploy/troubleshoot, prompt agent, managed harness agent, Copilot harness, voice agent, create/update/invoke/evaluate agent, add tool to agent, agent insights, batch/continuous eval, continuous monitoring, agent CI/CD, optimize/improve prompt, optimize agent instructions, Agent Optimizer, dataset curation from traces, deploy model, model fine-tuning (SFT/DPO/RFT), Foundry project, RBAC, role assignment, permissions, quota, capacity, region, deployment failure, AI Services, create Foundry resource, knowledge index, customize deployment, onboard, availability, training-data, grader, distillation, large file upload. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--microsoft-foundry at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 151

foundry-agent/eval-datasets/references/dataset-comparison.md

Dataset Comparison — A/B Testing Across Dataset Versions

Run structured experiments that compare how an agent performs across different dataset versions, and present results as leaderboards with per-evaluator breakdowns. Use this to answer: "Did scores drop because of harder tests or agent regression?"

Experiment Structure

An experiment consists of:

  1. Pinned agent version — the same agent evaluated on each dataset
  2. Varied dataset versions — the versions being compared
  3. Same evaluators — applied consistently across all runs
  4. Comparison results — which dataset version the agent performs better on

Step 1 — Define the Experiment

| Parameter | Value | Example | |-----------|-------|---------| | Agent | Pinned agent version | v3 | | Baseline dataset | Previous dataset version | support-bot-prod-traces-v2 | | Treatment dataset(s) | New dataset version(s) | support-bot-prod-traces-v3 | | Evaluators | Same set for all runs | coherence, fluency, relevance, intent_resolution, task_adherence |

Step 2 — Run Evaluations

For each dataset version, run evaluation_agent_batch_eval_create with:

  • Same evaluationId (groups all runs for comparison)
  • Same agentVersion
  • Same evaluatorNames
  • Different inputData (from each dataset version)

Important: Use evaluationId on evaluation_agent_batch_eval_create to group runs. After the runs exist, switch to evalId for evaluation_get and evaluation_comparison_create.

⚠️ Eval-group immutability: Keep the evaluator set and thresholds fixed within one evaluation group. If you need to change evaluators or thresholds, create a new evaluation group instead of reusing the previous evaluationId.

⚠️ Score drops are expected. When comparing v1→v2 datasets, lower scores on the new dataset likely mean the new test cases are harder (better coverage), not that the agent regressed. Do NOT remove dataset rows or weaken evaluators to recover scores. Instead, optimize the agent for the new failure patterns, then re-evaluate.

Step 3 — Compare Results

Use evaluation_comparison_create with the baseline and treatment runs:

{
  "insightRequest": {
    "displayName": "Dataset comparison: traces-v2 vs traces-v3 on agent-v3",
    "state": "NotStarted",
    "request": {
      "type": "EvaluationComparison",
      "evalId": "<eval-group-id>",
      "baselineRunId": "<traces-v2-run-id>",
      "treatmentRunIds": ["<traces-v3-run-id>"]
    }
  }
}

⚠️ Common mistake: evaluation_comparison_create uses insightRequest.request.evalId, not evaluationId, even when the runs were originally grouped with evaluationId.

Step 4 — Leaderboard

Present results as a leaderboard table:

| Evaluator | traces-v2 (baseline) | traces-v3 | Effect | |-----------|:---:|:---:|:---:| | Coherence | 4.0 | 3.6 | ⚠️ Lower | | Fluency | 4.5 | 4.3 | ⚠️ Lower | | Relevance | 3.6 | 3.2 | ⚠️ Lower | | Intent Resolution | 4.1 | 3.7 | ⚠️ Lower | | Task Adherence | 3.9 | 3.4 | ⚠️ Lower |

Recommendation

If scores drop uniformly across all evaluators, the new dataset is likely harder:

"Agent v3 scores dropped on traces-v3 across all evaluators. traces-v3 added 15 edge-case queries from production failures. This is expected — optimize the agent for the new failure patterns rather than reverting the dataset."

Pairwise A/B Comparison

For detailed pairwise analysis between exactly two dataset versions:

| Evaluator | Baseline (traces-v2) | Treatment (traces-v3) | Delta | p-value | Effect | |-----------|:---:|:---:|:---:|:---:|:---:| | Coherence | 4.0 ± 0.6 | 3.6 ± 0.9 | −0.4 | 0.03 | Degraded | | Fluency | 4.5 ± 0.4 | 4.3 ± 0.5 | −0.2 | 0.12 | Inconclusive | | Relevance | 3.6 ± 0.9 | 3.2 ± 1.1 | −0.4 | 0.04 | Degraded |

💡 Tip: The evaluation_comparison_create result includes pValue and treatmentEffect fields. Use pValue < 0.05 as the threshold for statistical significance.

Multi-Dataset Comparison

Compare how the same agent version performs across different datasets:

| Dataset | Coherence | Fluency | Relevance | Notes | |---------|:---------:|:-------:|:---------:|-------| | traces-v3 (prod) | 4.0 | 4.5 | 3.6 | Production-derived | | synthetic-v2 | 4.3 | 4.6 | 4.1 | May overestimate quality | | manual-v1 (curated) | 3.8 | 4.4 | 3.2 | Hardest test cases |

⚠️ Warning: Be cautious comparing scores across datasets with different structures (e.g., production traces vs synthetic). Differences may reflect dataset difficulty, not agent quality.

Next Steps