AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

finetuning

Fine-tune models on Microsoft Foundry using SFT (supervised), DPO (preference), or RFT (reinforcement with graders). Covers dataset preparation, training job submission, deployment, and evaluation. USE FOR: fine-tune, SFT, DPO, RFT, training data, grader, distillation, fine-tuned model, training job, large file upload, calibrate grader, deploy fine-tuned model, evaluate fine-tuned model. DO NOT USE FOR: general model deployment without fine-tuning (use deploy-model), agent creation (use agents), prompt optimization without training (use prompt-optimizer).

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--finetuning at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 18

references/vision-fine-tuning.md

Vision Fine-Tuning

Fine-tune models with image data to customize visual understanding. Uses the same chat-completions JSONL format as text SFT, but with image content blocks in user messages.

Supported Models

| Model | Version | |-------|---------| | gpt-4o | 2024-08-06 | | gpt-4.1 | 2025-04-14 |

Image Requirements

| Constraint | Limit | |-----------|-------| | Max examples with images per training file | 50,000 | | Max images per example | 64 | | Max image file size | 10 MB | | Supported formats | JPEG, PNG, WEBP | | Color mode | RGB or RGBA | | Min examples | 10 |

Important: Images can only appear in user messages, never in assistant responses.

Data Format

Each training example follows the standard SFT messages format. Images are included as image_url content blocks within user messages.

{"messages": [{"role": "system", "content": "You are a helpful AI assistant that describes images."}, {"role": "user", "content": [{"type": "text", "text": "Describe this image."}, {"type": "image_url", "image_url": {"url": "https://example.com/photo.png", "detail": "high"}}]}, {"role": "assistant", "content": "The image shows a cityscape with tall buildings against a blue sky."}]}

Image Sources

Images can be provided in two ways:

1. Public URL:

{"type": "image_url", "image_url": {"url": "https://example.com/image.png"}}

2. Base64 data URI:

{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}}

Detail Control

The detail parameter controls image processing fidelity and cost:

| Value | Behavior | Cost | |-------|----------|------| | low | Downscales to 512×512 pixels | Lower | | high | Full resolution processing | Higher | | auto | Model decides based on image size | Default |

{"type": "image_url", "image_url": {"url": "https://example.com/image.png", "detail": "low"}}

Use low for tasks where fine visual detail doesn't matter (classification, general description). Use high for tasks needing precise detail (OCR, diagram reading, defect detection).

Content Moderation

Images are screened before training. The following are automatically excluded:

  • Images containing people or faces (face detection only — no identification)
  • CAPTCHAs
  • Content violating Azure usage policies

This screening may add latency to file upload validation.

Best Practices

  • Diverse examples: Vary image content, angles, lighting, and resolution
  • Consistent annotations: Keep assistant response style and detail level uniform
  • Start with detail: low: Cheaper and faster — upgrade to high only if results need it
  • Check for excluded images: After upload, verify the training count matches expectations — some images may be silently skipped due to content moderation
  • Mixed text+image: You can include both text-only and image examples in the same training file

Training Workflow

Vision fine-tuning follows the exact same workflow as text SFT:

  1. Prepare JSONL with image content blocks
  2. Upload training file (validation may take longer due to image screening)
  3. Create fine-tuning job with a supported vision model
  4. Monitor and evaluate as usual
# Upload (image validation may take longer)
train_file = client.files.create(purpose="fine-tune", file=open("vision_train.jsonl", "rb"))
client.files.wait_for_processing(train_file.id)

# Submit — same as text SFT
job = client.fine_tuning.jobs.create(
    model="gpt-4.1-2025-04-14",
    training_file=train_file.id,
    validation_file=val_file.id,
    method={"type": "supervised"}
)

Troubleshooting

| Issue | Resolution | |-------|-----------| | Images skipped silently | Check for people/faces, oversized files, unsupported formats | | URL not accessible | Ensure URLs are publicly accessible, or use base64 data URIs | | Exceeds 10 MB | Resize or compress the image | | Wrong color mode | Convert to RGB or RGBA | | Low quality results | Try detail: high, add more diverse examples, increase dataset size |

Reference