AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

phoenix-evals

Build and run evaluators for AI/LLM applications using Phoenix.

Original source
github
License
MIT
Pinned revision
5269f7ea2d6c3a9bfda5ed878b6726cf509121fd

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read github--phoenix-evals at revision 5269f7ea2d6c3a9bfda5ed878b6726cf509121fd, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 35

references/experiments-overview.md

Experiments: Overview

Systematic testing of AI systems with datasets, tasks, and evaluators.

Structure

DATASET     → Examples: {input, expected_output, metadata}
TASK        → function(input) → output
EVALUATORS  → (input, output, expected) → score
EXPERIMENT  → Run task on all examples, score results

Basic Usage

from phoenix.client import Client

client = Client()
experiment = client.experiments.run_experiment(
    dataset=my_dataset,
    task=my_task,
    evaluators=[accuracy, faithfulness],
    experiment_name="improved-retrieval-v2",
)

print(experiment.aggregate_scores)
# {'accuracy': 0.85, 'faithfulness': 0.92}

Workflow

  1. Create dataset - From traces, synthetic data, or manual curation
  2. Define task - The function to test (your LLM pipeline)
  3. Select evaluators - Code and/or LLM-based
  4. Run experiment - Execute and score
  5. Analyze & iterate - Review, modify task, re-run

Dry Runs

Test setup before full execution:

experiment = client.experiments.run_experiment(
    dataset=dataset,
    task=task,
    evaluators=evaluators,
    dry_run=3,
)  # Just 3 examples

Async Usage

Use AsyncClient when your task or evaluators make network calls and you want higher throughput:

from phoenix.client import AsyncClient

client = AsyncClient()
experiment = await client.experiments.run_experiment(
    dataset=my_dataset,
    task=my_async_task,
    evaluators=[accuracy, faithfulness],
    experiment_name="improved-retrieval-v2",
)

Best Practices

  • Name meaningfully: "improved-retrieval-v2-2024-01-15" not "test"
  • Version datasets: Don't modify existing
  • Multiple evaluators: Combine perspectives