AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

microsoft-foundry

Build, deploy, evaluate, optimize, fine-tune, and manage Microsoft Foundry agents, models, and resources end to end. USE FOR: foundry, azd ai agent, azd provision/deploy, hosted agent scaffold/develop/run/deploy/troubleshoot, prompt agent, managed harness agent, Copilot harness, voice agent, create/update/invoke/evaluate agent, add tool to agent, agent insights, batch/continuous eval, continuous monitoring, agent CI/CD, optimize/improve prompt, optimize agent instructions, Agent Optimizer, dataset curation from traces, deploy model, model fine-tuning (SFT/DPO/RFT), Foundry project, RBAC, role assignment, permissions, quota, capacity, region, deployment failure, AI Services, create Foundry resource, knowledge index, customize deployment, onboard, availability, training-data, grader, distillation, large file upload. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--microsoft-foundry at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 151

foundry-agent/eval-datasets/references/eval-trending.md

Eval Trending — Metrics Over Time

Track evaluation metrics across multiple runs and versions to visualize improvement trends and detect regressions. This addresses the gap of understanding how agent quality changes over time.

Prerequisites

  • At least 2 evaluation runs in the same evaluation group (same evaluationId when created)
  • Project endpoint and selected environment resolved from azd or the selected metadata overlay

⚠️ Eval-group immutability: Trend a group only when its evaluator set and thresholds stayed fixed across runs. If either changed, start a new evaluation group and track that history separately.

Step 1 — Retrieve Evaluation History

Use evaluation_get to list all evaluation groups:

| Parameter | Required | Description | |-----------|----------|-------------| | projectEndpoint | ✅ | Azure AI Project endpoint | | isRequestForRuns | | false (default) to list evaluation groups |

Then retrieve all runs within the target evaluation group:

| Parameter | Required | Description | |-----------|----------|-------------| | projectEndpoint | ✅ | Azure AI Project endpoint | | evalId | ✅ | Evaluation group ID | | isRequestForRuns | ✅ | true to list runs |

⚠️ Parameter guardrail: evaluation_get expects evalId, not evaluationId, even if the runs were grouped earlier with evaluationId.

Step 2 — Build Metrics Timeline

For each run, extract per-evaluator scores and build a timeline:

| Run | Agent Version | Date | Coherence | Fluency | Relevance | Intent Resolution | Task Adherence | Safety | |-----|--------------|------|-----------|---------|-----------|-------------------|----------------|--------| | run-001 | v1 | 2025-01-15 | 3.2 | 4.1 | 2.8 | 3.0 | 2.5 | 0.95 | | run-002 | v2 | 2025-01-22 | 3.8 | 4.3 | 3.5 | 3.7 | 3.2 | 0.97 | | run-003 | v3 | 2025-02-01 | 4.1 | 4.4 | 4.0 | 4.2 | 3.8 | 0.96 | | run-004 | v4 | 2025-02-08 | 4.0 | 4.5 | 3.6 | 4.1 | 3.9 | 0.98 |

Step 3 — Trend Analysis

Calculate trends for each evaluator:

| Evaluator | v1 → v4 Change | Trend | Status | |-----------|----------------|-------|--------| | Coherence | +0.8 (+25%) | ↑ Improving | ✅ | | Fluency | +0.4 (+10%) | ↑ Improving | ✅ | | Relevance | +0.8 (+29%) | ↑ Improving (dip at v4) | ⚠️ | | Intent Resolution | +1.1 (+37%) | ↑ Improving | ✅ | | Task Adherence | +1.4 (+56%) | ↑ Improving | ✅ | | Safety | +0.03 (+3%) | → Stable | ✅ |

Detecting Regressions

Flag any evaluator where the latest run scored lower than the previous run:

| Evaluator | Previous (v3) | Latest (v4) | Delta | Alert | |-----------|--------------|-------------|-------|-------| | Relevance | 4.0 | 3.6 | -0.4 (-10%) | ⚠️ REGRESSION |

⚠️ Regression detected: Relevance dropped 10% from v3 to v4. Investigate prompt changes or dataset drift. See Eval Regression for automated analysis.

Trend Visualization (Text-based)

Coherence   ████████████████████████████████░░░░░░ 4.0/5.0  ↑ +25%
Fluency     █████████████████████████████████████░░ 4.5/5.0  ↑ +10%
Relevance   ████████████████████████████░░░░░░░░░░ 3.6/5.0  ↑ +29% ⚠️ dip
Intent Res. █████████████████████████████████░░░░░░ 4.1/5.0  ↑ +37%
Task Adh.   ████████████████████████████████░░░░░░░ 3.9/5.0  ↑ +56%
Safety      ████████████████████████████████████████ 0.98     → Stable

Step 4 — Cross-Version Summary

Present an executive summary:

"Over 4 agent versions (v1→v4), your agent has improved significantly across all quality metrics. The biggest gain is Task Adherence (+56%). However, Relevance showed a 10% regression from v3 to v4 — recommend investigating recent prompt changes. Safety remains stable at 98%."

Recommended Thresholds

| Severity | Threshold | Action | |----------|-----------|--------| | ✅ Healthy | ≤ 2% drop from previous run | No action needed | | ⚠️ Warning | 2–5% drop from previous run | Review recent changes | | 🔴 Regression | > 5% drop from previous run | Block deployment, investigate | | 🔴 Critical | Below baseline (v1) on any metric | Rollback to last known good version |

Next Steps