AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

phoenix-evals

Build and run evaluators for AI/LLM applications using Phoenix.

Original source
github
License
MIT
Pinned revision
5269f7ea2d6c3a9bfda5ed878b6726cf509121fd

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read github--phoenix-evals at revision 5269f7ea2d6c3a9bfda5ed878b6726cf509121fd, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 35

references/fundamentals.md

Fundamentals

Application-specific tests for AI systems. Code first, LLM for nuance, human for truth.

Evaluator Types

| Type | Speed | Cost | Use Case | | ---- | ----- | ---- | -------- | | Code | Fast | Cheap | Regex, JSON, format, exact match | | LLM | Medium | Medium | Subjective quality, complex criteria | | Human | Slow | Expensive | Ground truth, calibration |

Decision: Code first → LLM only when code can't capture criteria → Human for calibration.

Score Structure

| Property | Required | Description | | -------- | -------- | ----------- | | name | Yes | Evaluator name | | kind | Yes | "code", "llm", "human" | | score | No* | 0-1 numeric | | label | No* | "pass", "fail" | | explanation | No | Rationale |

*One of score or label required.

Binary > Likert

Use pass/fail, not 1-5 scales. Clearer criteria, easier calibration.

# Multiple binary checks instead of one Likert scale
evaluators = [
    AnswersQuestion(),    # Yes/No
    UsesContext(),        # Yes/No
    NoHallucination(),    # Yes/No
]

Quick Patterns

Code Evaluator

from phoenix.evals import create_evaluator

@create_evaluator(name="has_citation", kind="code")
def has_citation(output: str) -> bool:
    return bool(re.search(r'\[\d+\]', output))

LLM Evaluator

from phoenix.evals import ClassificationEvaluator, LLM

evaluator = ClassificationEvaluator(
    name="helpfulness",
    prompt_template="...",
    llm=LLM(provider="openai", model="gpt-4o"),
    choices={"not_helpful": 0, "helpful": 1}
)

Run Experiment

from phoenix.client.experiments import run_experiment

experiment = run_experiment(
    dataset=dataset,
    task=my_task,
    evaluators=[evaluator1, evaluator2],
)
print(experiment.aggregate_scores)