AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

microsoft-foundry

Build, deploy, evaluate, optimize, fine-tune, and manage Microsoft Foundry agents, models, and resources end to end. USE FOR: foundry, azd ai agent, azd provision/deploy, hosted agent scaffold/develop/run/deploy/troubleshoot, prompt agent, managed harness agent, Copilot harness, voice agent, create/update/invoke/evaluate agent, add tool to agent, agent insights, batch/continuous eval, continuous monitoring, agent CI/CD, optimize/improve prompt, optimize agent instructions, Agent Optimizer, dataset curation from traces, deploy model, model fine-tuning (SFT/DPO/RFT), Foundry project, RBAC, role assignment, permissions, quota, capacity, region, deployment failure, AI Services, create Foundry resource, knowledge index, customize deployment, onboard, availability, training-data, grader, distillation, large file upload. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare).

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--microsoft-foundry at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 151

foundry-agent/create/create-voice.md

Create a Microsoft Foundry Voice Agent

Create, configure, or extend a Microsoft Foundry Voice Agent:

  • Prompt Voice: one managed Voice Agent using modelType: managed or self_deployed.
  • Hosted Voice: a voice-based agent using a hosted text agent as its conversation engine, with a managed Voice wrapper (kind: voice) and hosted target (kind: hosted).

For a custom Hosted WebSocket agent with developer-defined frames and no managed Voice wrapper, use invocations-ws, not this workflow.

Quick Reference

| Decision | Prompt Voice | Hosted Voice | |---|---|---| | Foundry agents | One kind: voice service | One hosted target plus one Voice wrapper | | Reasoning and tools | Voice service | Hosted target | | Audio, VAD, TTS, greeting | Voice service | Voice wrapper | | Local runtime | None | Target-only Bridge validation | | End-to-end invocation | Managed voice WebSocket | Managed voice WebSocket on the wrapper |

Step 1: Prerequisite and Context Resolution

Before authoring files or running azd:

  1. From the foundry-agent/create directory, run the paired preflight scripts:

    ./scripts/check-copilot-app-entry.sh
    ./scripts/verify-environment.sh
    
    .\scripts\check-copilot-app-entry.ps1
    .\scripts\verify-environment.ps1
    

    Resolve every [ACTION] before continuing. Never run az login or azd auth login for the user.

  2. Read Foundry azd guidance before any azd command.

    Commands below use the repository's neutral azd ... form. Execute every azd invocation with the user agent set inline:

    AZURE_DEV_USER_AGENT=microsoft_foundry_skill azd <arguments>
    
    $env:AZURE_DEV_USER_AGENT = "microsoft_foundry_skill"
    azd <arguments>
    

    Do not persist AZURE_DEV_USER_AGENT with azd env set, .env, or azure.yaml.

  3. Follow Common Project Context Resolution:

    • Inspect azure.yaml and azd env get-values when an azd project exists.
    • Classify existing services before selecting a target. Do not convert an existing Prompt, Hosted, Managed Harness, or Voice Agent into another type.
    • If multiple services exist, ask which Voice wrapper or Prompt Voice service to modify. Selecting a Voice wrapper also selects its conversationEngine.name hosted target as a deployment dependency.
    • Resolve an existing project endpoint to its ARM ID with scripts/resolve-project-id.sh or scripts/resolve-project-id.ps1; never construct the ARM ID from the endpoint.

Step 2: Model-First Decision and Collected-Values Checkpoint

Before generating azure.yaml, scaffolding code, or running init, determine the conversation engine and model strategy from the request and existing project context.

For a generic greenfield request to create a Voice Agent, proceed with Prompt Voice using the managed real-time model without requiring a subtype question. State these defaults before authoring:

  • modelType: managed
  • model.id: gpt-realtime
  • canonical authored kind: voice

Resolve model strategy from deployment intent, not from the presence of a model name:

  • A model name alone, including a model name plus a project endpoint or name, uses managed Prompt Voice (modelType: managed) and does not create a project model deployment.
  • Explicit lifecycle wording selects self-deployed Prompt Voice: BYOM, self-deployed, project model deployment, existing deployment, deployment name, create/deploy/provision the model, create if missing/not found, ensure the deployment exists, or inspect/select its version, SKU, capacity, or quota. For example, "Use gpt-realtime; create it if it does not exist" is sufficient self-deployed intent.
  • Explicit user choice overrides inference. Ask one focused clarification only when signals conflict, such as requesting both managed/service-hosted behavior and creation or provisioning of a deployment.

Before authoring, state the resolved interpretation. For managed intent, state that no project deployment will be created. For self-deployed provisioning intent, state that the workflow will verify the deployment, add it to services.<ai-project-service>.deployments[] if missing, provision it through azd, and use modelType: self_deployed.

Ask only when another unresolved requirement materially changes the architecture:

  • custom code, an existing hosted text agent, or explicit Hosted Voice intent;
  • an existing project or codebase whose services imply a subtype but do not identify the target clearly;
  • a planning, explanation, or comparison request that explicitly asks to evaluate the available approaches.

For planning or comparison, explain Prompt Voice managed, Prompt Voice self-deployed, Hosted Voice, and custom raw invocations_ws without creating files or running azd.

Detect requested Voice Agent settings or existing settings that must be preserved, including structured inputs, audio, transcription, turn detection, voice, greeting/avatar, tools, handoff, response options, storage, or telephony. Do not ask about settings the user did not request.

When Twilio telephony is explicitly requested, follow the remote connection and binding flow in Voice Agent YAML settings. Ask only for a missing connection name, Account SID, Auth Token availability, or Twilio number; never ask for the token in chat.

For explicit ACS or Teams Phone extensibility (TPE) requests, follow the TPE/ACS section in Voice Agent YAML settings; do not automate infrastructure, RBAC, webhooks, or Event Grid.

Resolve values from the request, workspace, azure.yaml, and active azd environment before asking. Do not author files until this checkpoint is complete:

| Value | Resolution rule | |---|---| | Voice subtype | Default a generic greenfield request to managed Prompt Voice; ask only for the architecture-changing cases above | | Agent name | Reuse the selected service name; otherwise collect or derive a concise name | | Foundry project | Confirm new versus existing; never provision implicitly | | Existing project identity | Resolve both endpoint and ARM ID; pass the ARM ID to --project-id | | azd environment | Reuse the selected environment or collect the intended environment | | Model strategy | Use managed for a model name alone; use self-deployed for explicit deployment lifecycle intent; clarify only conflicting signals | | Model/deployment name | Default managed Prompt Voice to gpt-realtime; for self-deployed Prompt Voice, inspect the selected Foundry account for the exact deployment name before authoring | | Instructions | Preserve existing instructions or collect the requested behavior | | Output voice | Preserve an existing value or use the user's requested voice; otherwise omit it and use the azd default |

State every applied default clearly before authoring.

Step 3: Load Only the Selected Authoring Reference

Always read Voice Agent common invariants.

Do not load the other subtype's YAML reference unless the user asks for a comparison or migration.

Before editing settings, map every requested field to the Prompt Voice service, Voice wrapper, or hosted target using the settings ownership table. Preserve unrelated existing fields and edit only the requested YAML paths.

Step 4: Initialize or Update

Prompt Voice

For a new project:

azd ai agent init --no-prompt \
  --kind prompt-voice \
  --agent-name "<agent-name>" \
  --model "<model-name>"

For an existing Foundry project:

azd ai agent init --no-prompt \
  --kind prompt-voice \
  --agent-name "<agent-name>" \
  --project-id "<project-arm-id>" \
  --model "<model-name>"

Add --voice "<voice-name>" only when the user explicitly requests a voice. Otherwise allow azd to apply its default. --kind prompt-voice does not accept Hosted/runtime flags or --model-deployment. For a new self-deployed Prompt Voice service, initialize once, then follow the Golden Path below and Prompt Voice YAML.

Immediately after init, normalize the generated service to canonical kind: voice if azd wrote kind: prompt-voice, alongside the other generated-YAML edits.

For modelType: self_deployed, follow this Golden Path:

  1. Inspect the selected Foundry account for the exact deployment name. If a compatible deployment exists, set model.id to that exact name and do not add a duplicate deployment entry.
  2. If it is missing and the user intends creation, read Foundry Model Reference, resolve the exact model version, format, supported SKU/capacity, lifecycle, and quota, and state the planned billable/quota-consuming deployment. If creation was not explicit, obtain confirmation.
  3. Preserve existing project infrastructure and deployment entries, append the deployment under services.<ai-project-service>.deployments[], and set model.id to the exact deployments[].name. Use this path for both realtime and cascaded/text-model BYOM; Foundry infers the pipeline from model.id.
  4. Run azd provision --preview --no-prompt, confirm the preview changes only the intended model infrastructure, then run azd provision --no-prompt. Deploy the Voice service only after provisioning succeeds.

For an existing Prompt Voice service or an added model in an existing azd project, edit azure.yaml in place. Do not rerun init and create a suffixed duplicate. Do not use az cognitiveservices account deployment create; azure.yaml is the source of truth for azd-managed model infrastructure.

Hosted Voice

Use the official Hosted Voice manifest when starting from the sample:

azd ai agent init --no-prompt \
  -m "https://github.com/microsoft-foundry/foundry-samples/blob/main/samples/python/hosted-agents/bring-your-own/voice-agent-target-agent/basic/azure.yaml"

Add --project-id "<project-arm-id>" when using an existing project. For existing target code, preserve the target service and add or update the wrapper using Hosted Voice YAML; do not replace it with a Prompt Voice definition.

After init, enter the generated project directory before setting subscription, region, project name, endpoint, or model values. Do not chain azd env set to the init command.

Step 5: Local Validation

  • Prompt Voice has no local container. Continue to deploy.
  • Hosted Voice must validate the target's Bridge handler locally before deployment. Follow local run and Hosted Voice Bridge 1.0. Target-only validation does not replace an end-to-end wrapper smoke test.

Step 6: Deploy and Verify

Follow deploy, then invoke.

Voice services do not support azd ai agent invoke. Use the two-stage process in Voice smoke testing:

  1. Run the automated managed Voice WebSocket smoke test as the technical deployment gate.
  2. From azd ai agent show --output json, provide the Voice Agent Playground URL and recommend that the user complete manual audio-to-audio experience validation in the Foundry portal.

Run the helper from the microsoft-foundry skill root:

./foundry-agent/create/scripts/smoke-test-voice.sh \
  --endpoint "$AZURE_AI_PROJECT_ENDPOINT" \
  --agent-name "<voice-agent-name>" \
  --text "Hello, briefly introduce yourself." \
  --timeout 180
.\foundry-agent\create\scripts\smoke-test-voice.ps1 `
  -Endpoint $env:AZURE_AI_PROJECT_ENDPOINT `
  -AgentName "<voice-agent-name>" `
  -Text "Hello, briefly introduce yourself." `
  -Timeout 180

The final handoff must distinguish the results: report whether automated technical verification passed, provide the Playground URL, and explain that microphone, STT quality, VAD/turn-taking, interruption, playback/voice quality, browser/device behavior, and perceived latency remain a manual portal acceptance step. Do not claim that manual validation was completed automatically.

Error Handling

| Error | Resolution | |---|---| | Project choice is unresolved | Stop and ask for a new or existing project before init or provision | | Existing endpoint has no ARM ID | Resolve it with the paired resolve-project-id script | | Init would duplicate an existing service | Edit the existing Voice service in place | | Prompt Voice model deployment is missing | Do not author model.id alone. Reuse a compatible existing deployment, or confirm creation and follow the deployments[] preview/provision Golden Path | | Hosted target fails locally | Fix the hosted target's Bridge handler before deploying the Voice wrapper | | azd ai agent invoke rejects Voice | Use the managed voice WebSocket helper or Foundry Voice playground |