AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

finetuning

Fine-tune models on Microsoft Foundry using SFT (supervised), DPO (preference), or RFT (reinforcement with graders). Covers dataset preparation, training job submission, deployment, and evaluation. USE FOR: fine-tune, SFT, DPO, RFT, training data, grader, distillation, fine-tuned model, training job, large file upload, calibrate grader, deploy fine-tuned model, evaluate fine-tuned model. DO NOT USE FOR: general model deployment without fine-tuning (use deploy-model), agent creation (use agents), prompt optimization without training (use prompt-optimizer).

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--finetuning at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 18

references/evaluation.md

Evaluation Methodology

Principles

  1. Always establish a baseline: Evaluate the base (un-tuned) model first. Without a baseline, you can't measure improvement.
  2. Use a held-out test set: Never evaluate on training or validation data. The model has seen those.
  3. Use the same test set for every model: This is the only way to compare results fairly.
  4. Use task-specific graders: Built-in generic evaluators (Coherence, Fluency) measure general quality and won't detect fine-tuning improvements. Use custom graders (Python, score model, string check) for task-specific evaluation.
  5. Measure cost alongside accuracy: Report completion tokens per response when comparing models or checkpoints. A model that achieves the same accuracy with fewer tokens is strictly better — cheaper inference and lower latency.

Two-Layer Evaluation Strategy

Use the Azure AI Evaluation SDK (azure-ai-evaluation) for all evaluation.

| Layer | Purpose | Grader Type | When | |-------|---------|-------------|------| | Task-specific (primary) | Measure FT improvement | AzureOpenAIScoreModelGrader, AzureOpenAIPythonGrader, AzureOpenAIStringCheckGrader | Every eval | | General quality (guardrail) | Verify model didn't degrade | CoherenceEvaluator, FluencyEvaluator | Spot-check only |

Generic built-in evaluators (Coherence, Fluency, TaskAdherence) are guardrails, not metrics — they often show no difference between base and fine-tuned models even when domain-specific evaluation reveals clear improvement.

Custom Graders (Primary FT Evaluation)

1. Score Model Grader (LLM judge with task-specific rubric)

Best for: subjective tasks (summarization, alignment, style).

from azure.ai.evaluation import AzureOpenAIScoreModelGrader

summarization_grader = AzureOpenAIScoreModelGrader(
    model_config=model_config,
    name="summarization_quality",
    prompt="""Rate this news summary on a scale of 1-5.

Article: {{item.article}}
Summary: {{sample.output_text}}

Criteria:
- Captures ALL key facts (who, what, when, where)
- No hallucinated information not in the article
- Concise (under 3 sentences)

Score 1: Missing key facts or hallucinations
Score 3: Captures main point but misses details
Score 5: Perfect summary — all facts, no extras, concise

Return ONLY a number 1-5.""",
    output_type="numeric",
    pass_threshold=3,
)

2. Python Grader (programmatic/exact-match evaluation)

Best for: code generation, math, entity extraction, structured output.

from azure.ai.evaluation import AzureOpenAIPythonGrader

entity_grader = AzureOpenAIPythonGrader(
    name="entity_extraction_accuracy",
    source="""
import json

def grade(item, sample):
    try:
        extracted = json.loads(sample["output_text"])
        reference = json.loads(item["ground_truth"])
    except (json.JSONDecodeError, KeyError):
        return {"score": 0, "reason": "Invalid JSON output"}

    required_keys = ["people", "organizations", "locations", "dates"]
    missing = [k for k in required_keys if k not in extracted]
    if missing:
        return {"score": 0.5, "reason": f"Missing keys: {missing}"}

    total, matched = 0, 0
    for key in required_keys:
        ref_set = set(str(v).lower() for v in reference.get(key, []))
        ext_set = set(str(v).lower() for v in extracted.get(key, []))
        total += len(ref_set)
        matched += len(ref_set & ext_set)

    score = matched / total if total > 0 else 1.0
    return {"score": score, "reason": f"{matched}/{total} entities matched"}
""",
    pass_threshold=0.7,
)

3. String Check Grader (pattern matching)

Best for: classification, format compliance, tool calling format.

from azure.ai.evaluation import AzureOpenAIStringCheckGrader

tool_format_grader = AzureOpenAIStringCheckGrader(
    name="tool_call_format",
    input="{{sample.output_text}}",
    operation="like",          # or "eq", "starts_with", "contains"
    reference="function_call",
    pass_threshold=1,
)

classification_grader = AzureOpenAIStringCheckGrader(
    name="classification_accuracy",
    input="{{sample.output_text}}",
    operation="eq",
    reference="{{item.expected_label}}",
    pass_threshold=1,
)

Running an Evaluation

The evaluate() function runs multiple graders over an entire dataset:

from azure.ai.evaluation import evaluate, F1ScoreEvaluator

result = evaluate(
    data="eval_data.jsonl",
    evaluators={
        "task_grader": my_custom_score_grader,   # primary
        "f1": F1ScoreEvaluator(),                 # token overlap
    },
    output_path="./eval_results.json",
)

for metric, value in result["metrics"].items():
    print(f"{metric}: {value}")

Test Set Design

  • Size: 30–100 examples is sufficient.
  • Diversity: Cover easy/medium/hard, edge cases, and different sub-categories.
  • Quality: Reference answers must be gold-standard correct. A wrong reference penalizes correct outputs.

Interpreting Results

| Score Type | Range | Meaning | |-----------|-------|---------| | AI quality (1–5) | 1–2 Poor, 3 Adequate, 4 Good, 5 Excellent | | | NLP (0–1) | <0.3 Wrong, 0.3–0.6 Partial, 0.6–0.8 Good, >0.8 Strong | |

With 50+ eval examples, a difference of ~0.3 points (on 1–5 scale) is usually meaningful.

Evaluating RFT Models

  1. Evaluate with a DIFFERENT rubric than the training grader — otherwise you measure overfitting to the grader.
  2. Use F1ScoreEvaluator for exact-match accuracy.
  3. Use SimilarityEvaluator to catch semantically correct but differently formatted answers.
  4. Compare against the base model, not just other fine-tunes.

Reference