AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

azure-ai-projects-py

Build AI applications using the Azure AI Projects Python SDK (azure-ai-projects). Use when working with Foundry project clients, creating versioned agents with PromptAgentDefinition, running evaluations, managing connections/deployments/datasets/indexes, or using OpenAI-compatible clients. This is the high-level Foundry SDK - for low-level agent operations, use azure-ai-agents-python skill.

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--azure-ai-projects-py at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 11

references/built-in-evaluators.md

Built-in Evaluators Reference

Complete reference for Microsoft Foundry's built-in evaluators using the azure-ai-projects SDK.

Discovering Evaluators

List All Built-in Evaluators

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]

with (
    DefaultAzureCredential() as credential,
    AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
):
    evaluators = project_client.evaluators.list_latest_versions(type="builtin")
    for e in evaluators:
        print(f"{e.name}: {e.description}")
        print(f"  Categories: {e.categories}")

Get Evaluator Schema

Before using an evaluator, query its schema to discover required inputs:

evaluator = project_client.evaluators.get_version(
    name="builtin.task_adherence",
    version="latest"
)
print(f"Init Parameters: {evaluator.definition.init_parameters}")
print(f"Data Schema: {evaluator.definition.data_schema}")
print(f"Metrics: {evaluator.definition.metrics}")

Using Built-in Evaluators

All built-in evaluators use the azure_ai_evaluator type with builtin. prefix:

testing_criteria = [
    {
        "type": "azure_ai_evaluator",
        "name": "my_coherence_check",          # Your custom name for results
        "evaluator_name": "builtin.coherence", # The actual evaluator
        "data_mapping": {
            "query": "{{item.query}}",
            "response": "{{item.response}}"
        },
        "initialization_parameters": {
            "deployment_name": "gpt-4o-mini"   # Required for LLM-based evaluators
        }
    }
]

Quality Evaluators

builtin.coherence

Measures logical flow and consistency of the response.

{
    "type": "azure_ai_evaluator",
    "name": "coherence",
    "evaluator_name": "builtin.coherence",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Inputs: query, response
Output: Score 1-5 (5 = highly coherent)

builtin.fluency

Measures grammatical correctness and natural language quality.

{
    "type": "azure_ai_evaluator",
    "name": "fluency",
    "evaluator_name": "builtin.fluency",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Inputs: query, response
Output: Score 1-5 (5 = perfectly fluent)

builtin.relevance

Measures how well the response addresses the query given context.

{
    "type": "azure_ai_evaluator",
    "name": "relevance",
    "evaluator_name": "builtin.relevance",
    "data_mapping": {
        "query": "{{item.query}}",
        "response": "{{item.response}}",
        "context": "{{item.context}}"
    },
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Inputs: query, response, context
Output: Score 1-5 (5 = highly relevant)

builtin.groundedness

Measures whether the response is factually grounded in the provided context.

{
    "type": "azure_ai_evaluator",
    "name": "groundedness",
    "evaluator_name": "builtin.groundedness",
    "data_mapping": {
        "query": "{{item.query}}",
        "response": "{{item.response}}",
        "context": "{{item.context}}"
    },
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Inputs: query, response, context
Output: Score 1-5 (5 = fully grounded)

builtin.response_completeness

Measures whether the response fully addresses all aspects of the query.

{
    "type": "azure_ai_evaluator",
    "name": "response_completeness",
    "evaluator_name": "builtin.response_completeness",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Inputs: query, response
Output: Score 1-5

Safety Evaluators

Safety evaluators detect harmful content. They don't require deployment_name.

builtin.violence

Detects violent content.

{
    "type": "azure_ai_evaluator",
    "name": "violence",
    "evaluator_name": "builtin.violence",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}

Inputs: query, response
Output: pass/fail with severity score

builtin.sexual

Detects inappropriate sexual content.

{
    "type": "azure_ai_evaluator",
    "name": "sexual",
    "evaluator_name": "builtin.sexual",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}

builtin.self_harm

Detects content promoting or describing self-harm.

{
    "type": "azure_ai_evaluator",
    "name": "self_harm",
    "evaluator_name": "builtin.self_harm",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}

builtin.hate_unfairness

Detects biased or hateful content.

{
    "type": "azure_ai_evaluator",
    "name": "hate_unfairness",
    "evaluator_name": "builtin.hate_unfairness",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}

Agent Evaluators

Agent evaluators assess AI agent behavior and tool usage.

builtin.task_adherence

Evaluates whether the agent follows its system instructions.

{
    "type": "azure_ai_evaluator",
    "name": "task_adherence",
    "evaluator_name": "builtin.task_adherence",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Note: Use {{sample.output_items}} for agent responses to include tool call information.

builtin.intent_resolution

Evaluates whether the agent correctly understood user intent.

{
    "type": "azure_ai_evaluator",
    "name": "intent_resolution",
    "evaluator_name": "builtin.intent_resolution",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_text}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

builtin.task_completion

Evaluates whether the agent completed the task end-to-end.

{
    "type": "azure_ai_evaluator",
    "name": "task_completion",
    "evaluator_name": "builtin.task_completion",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

builtin.tool_call_accuracy

Evaluates whether tool calls are correct (selection + parameters).

{
    "type": "azure_ai_evaluator",
    "name": "tool_call_accuracy",
    "evaluator_name": "builtin.tool_call_accuracy",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

builtin.tool_call_success

Evaluates whether tool calls executed without failures.

{
    "type": "azure_ai_evaluator",
    "name": "tool_call_success",
    "evaluator_name": "builtin.tool_call_success",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"}
}

builtin.tool_selection

Evaluates whether the correct tools were selected.

{
    "type": "azure_ai_evaluator",
    "name": "tool_selection",
    "evaluator_name": "builtin.tool_selection",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

NLP Evaluators

NLP evaluators compare responses to ground truth without requiring an LLM.

builtin.f1_score

Token-level F1 score between response and ground truth.

{
    "type": "azure_ai_evaluator",
    "name": "f1",
    "evaluator_name": "builtin.f1_score",
    "data_mapping": {"response": "{{item.response}}", "ground_truth": "{{item.ground_truth}}"}
}

Output: Score 0-1

builtin.bleu_score

BLEU score for generation quality.

{
    "type": "azure_ai_evaluator",
    "name": "bleu",
    "evaluator_name": "builtin.bleu_score",
    "data_mapping": {"response": "{{item.response}}", "ground_truth": "{{item.ground_truth}}"}
}

builtin.rouge_score

ROUGE score for summarization quality.

{
    "type": "azure_ai_evaluator",
    "name": "rouge",
    "evaluator_name": "builtin.rouge_score",
    "data_mapping": {"response": "{{item.response}}", "ground_truth": "{{item.ground_truth}}"}
}

builtin.similarity

Semantic similarity between response and ground truth.

{
    "type": "azure_ai_evaluator",
    "name": "similarity",
    "evaluator_name": "builtin.similarity",
    "data_mapping": {
        "query": "{{item.query}}",
        "response": "{{item.response}}",
        "ground_truth": "{{item.ground_truth}}"
    },
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Evaluator Sets by Use Case

Quick Health Check

testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "coherence", "evaluator_name": "builtin.coherence", ...},
    {"type": "azure_ai_evaluator", "name": "fluency", "evaluator_name": "builtin.fluency", ...},
    {"type": "azure_ai_evaluator", "name": "violence", "evaluator_name": "builtin.violence", ...},
]

Safety Audit

testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "violence", "evaluator_name": "builtin.violence", ...},
    {"type": "azure_ai_evaluator", "name": "sexual", "evaluator_name": "builtin.sexual", ...},
    {"type": "azure_ai_evaluator", "name": "self_harm", "evaluator_name": "builtin.self_harm", ...},
    {"type": "azure_ai_evaluator", "name": "hate_unfairness", "evaluator_name": "builtin.hate_unfairness", ...},
]

Agent Evaluation

testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "task_adherence", "evaluator_name": "builtin.task_adherence", ...},
    {"type": "azure_ai_evaluator", "name": "intent_resolution", "evaluator_name": "builtin.intent_resolution", ...},
    {"type": "azure_ai_evaluator", "name": "tool_call_accuracy", "evaluator_name": "builtin.tool_call_accuracy", ...},
]

RAG Evaluation

testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "groundedness", "evaluator_name": "builtin.groundedness", ...},
    {"type": "azure_ai_evaluator", "name": "relevance", "evaluator_name": "builtin.relevance", ...},
    {"type": "azure_ai_evaluator", "name": "response_completeness", "evaluator_name": "builtin.response_completeness", ...},
]

Data Mapping Reference

| Data Source | Response Mapping | Use Case | |-------------|------------------|----------| | JSONL dataset | {{item.response}} | Pre-recorded query/response pairs | | Agent target | {{sample.output_text}} | Plain text response | | Agent target | {{sample.output_items}} | Structured JSON with tool calls |

When to use sample.output_items:

  • Tool-related evaluators (tool_call_accuracy, tool_selection, etc.)
  • Task adherence evaluator
  • Any evaluator needing tool call context

Related Documentation