AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

microsoft-foundry

Build, deploy, evaluate, optimize, fine-tune, and manage Microsoft Foundry agents, models, and resources end to end. USE FOR: foundry, azd ai agent, azd provision/deploy, hosted agent scaffold/develop/run/deploy/troubleshoot, prompt agent, managed harness agent, Copilot harness, voice agent, create/update/invoke/evaluate agent, add tool to agent, agent insights, batch/continuous eval, continuous monitoring, agent CI/CD, optimize/improve prompt, optimize agent instructions, Agent Optimizer, dataset curation from traces, deploy model, model fine-tuning (SFT/DPO/RFT), Foundry project, RBAC, role assignment, permissions, quota, capacity, region, deployment failure, AI Services, create Foundry resource, knowledge index, customize deployment, onboard, availability, training-data, grader, distillation, large file upload. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--microsoft-foundry at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 151

foundry-agent/agent-optimizer/references/eval-yaml.md

eval.yaml Guidance

Create eval.yaml directly when the conversation or .foundry/agent-metadata*.yaml already selected the dataset/evaluators. Otherwise ask whether to run azd ai agent eval generate or let optimize use built-in defaults.

Include

name: <suite-or-optimization-name>
agent:
  name: <agent-name>
  kind: hosted
  version: "<agent-version>"
  model: <baseline-model-deployment-name>
  config: .agent_configs/baseline/metadata.yaml
dataset:
  local_uri: <path-to-jsonl>
  # name: <foundry-dataset-name>
  # version: "<dataset-version>"
# validation_dataset:
#   name: <validation-dataset-name>
#   version: "<validation-version>"
evaluators:
  - <evaluator-name>
  - name: <custom-evaluator-name>
    version: "<evaluator-version>"
    local_uri: <local-evaluator-json>
options:
  eval_model: <existing-chat-model-deployment-name>
  optimization_model: <allowed-optimizer-model-deployment-name>
  max_candidates: 4
  optimization_config:
    model_search_space:
      - <target-model-deployment-name>

Use existing model deployments for agent.model and options.eval_model; do not assume gpt-4o.

For options.optimization_model, first verify that the target Foundry project has a deployment whose name is in this allowlist:

  • GPT-5
  • GPT-5.1
  • GPT-5.2
  • GPT-5.4
  • GPT-5.5
  • DeepSeek-V4-Pro
  • DeepSeek-V-3.2

If none exist, ask the user to deploy one before configuring optimization. Use options.optimization_config.model_search_space only for target model candidates that exist in the project; it may include the baseline model when the user wants it compared.

Generate evals when inputs are missing

Prefer eval generate over older init flows:

azd ai agent eval generate --dataset <path-to-jsonl>
azd ai agent eval generate --reset-defaults

After generation, run azd ai agent optimize --optimize-model <allowed-optimizer-model-deployment-name> from the azd project; optimize auto-detects the generated eval.yaml.

Skip

Do not add these fields unless the user explicitly asks and understands the tradeoff:

  • target_attributes
  • budget
  • min_improvement
  • pass_threshold
  • keep_versions
  • generation_instruction
  • max_samples
  • trace_days
  • legacy dataset_file, dataset_reference, or validation_reference when writing a new file

Keep target_attributes omitted so azd can auto-detect optimizable attributes.

Source mapping

| Source | eval.yaml field | |--------|-----------------| | effective azd context | agent.name, agent.version, agent.kind | | baseline config | agent.model, agent.config | | selected local dataset JSONL | dataset.local_uri | | selected remote/local dataset | dataset.name, dataset.version, dataset.local_uri | | selected validation dataset | validation_dataset | | selected Foundry/local evaluators | evaluators[] | | selected judge/eval deployment | options.eval_model | | selected optimizer deployment | options.optimization_model | | selected target model candidates | options.optimization_config.model_search_space |

Treat older dataset_file, dataset_reference, validation_reference, max_iterations, and optimization_config.model as legacy inputs when reading existing files, but write new files with the current contract above.