AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

phoenix-evals

Build and run evaluators for AI/LLM applications using Phoenix.

Original source
github
License
MIT
Pinned revision
5269f7ea2d6c3a9bfda5ed878b6726cf509121fd

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read github--phoenix-evals at revision 5269f7ea2d6c3a9bfda5ed878b6726cf509121fd, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 35

references/fundamentals-anti-patterns.md

Anti-Patterns

Common mistakes and fixes.

| Anti-Pattern | Problem | Fix | | ------------ | ------- | --- | | Generic metrics | Pre-built scores don't match your failures | Build from error analysis | | Vibe-based | No quantification | Measure with experiments | | Ignoring humans | Uncalibrated LLM judges | Validate >80% TPR/TNR | | Premature automation | Evaluators for imagined problems | Let observed failures drive | | Saturation blindness | 100% pass = no signal | Keep capability evals at 50-80% | | Similarity metrics | BERTScore/ROUGE for generation | Use for retrieval only | | Model switching | Hoping a model works better | Error analysis first | | Single-run scoring | LLM judges and non-deterministic tasks add per-run noise that can drown the signal from a prompt change on a small dataset | Set repetitions on runExperiment (or grow the dataset) when the task or judge is an LLM call |

Quantify Changes

from phoenix.client import Client

client = Client()
baseline = client.experiments.run_experiment(dataset=dataset, task=old_prompt, evaluators=evaluators)
improved = client.experiments.run_experiment(dataset=dataset, task=new_prompt, evaluators=evaluators)
print(f"Improvement: {improved.pass_rate - baseline.pass_rate:+.1%}")

Don't Use Similarity for Generation

# BAD
score = bertscore(output, reference)

# GOOD
correct_facts = check_facts_against_source(output, context)

Error Analysis Before Model Change

# BAD
for model in models:
    results = test(model)

# GOOD
failures = analyze_errors(results)
# Then decide if model change is warranted