AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

phoenix-evals

Build and run evaluators for AI/LLM applications using Phoenix.

Original source
github
License
MIT
Pinned revision
5269f7ea2d6c3a9bfda5ed878b6726cf509121fd

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read github--phoenix-evals at revision 5269f7ea2d6c3a9bfda5ed878b6726cf509121fd, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 35

references/fundamentals-model-selection.md

Model Selection

Error analysis first, model changes last.

Decision Tree

Performance Issue?
       │
       ▼
Error analysis suggests model problem?
    NO  → Fix prompts, retrieval, tools
    YES → Is it a capability gap?
          YES → Consider model change
          NO  → Fix the actual problem

Judge Model Selection

| Principle | Action | | --------- | ------ | | Start capable | Use gpt-4o first | | Optimize later | Test cheaper after criteria stable | | Same model OK | Judge does different task |

# Start with capable model
judge = ClassificationEvaluator(
    llm=LLM(provider="openai", model="gpt-4o"),
    ...
)

# After validation, test cheaper
judge_cheap = ClassificationEvaluator(
    llm=LLM(provider="openai", model="gpt-4o-mini"),
    ...
)
# Compare TPR/TNR on same test set

Don't Model Shop

from phoenix.client import Client

client = Client()

# BAD
for model in ["gpt-4o", "claude-3", "gemini-pro"]:
    results = client.experiments.run_experiment(
        dataset=dataset,
        task=lambda input, _model=model: task(input, model=_model),
        evaluators=evaluators,
    )

# GOOD
failures = analyze_errors(results)
# "Ignores context" → Fix prompt
# "Can't do math" → Maybe try better model

When Model Change Is Warranted

  • Failures persist after prompt optimization
  • Capability gaps (reasoning, math, code)
  • Error analysis confirms model limitation