AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

phoenix-evals

Build and run evaluators for AI/LLM applications using Phoenix.

Original source
github
License
MIT
Pinned revision
5269f7ea2d6c3a9bfda5ed878b6726cf509121fd

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read github--phoenix-evals at revision 5269f7ea2d6c3a9bfda5ed878b6726cf509121fd, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 35

references/production-overview.md

Production: Overview

CI/CD evals vs production monitoring - complementary approaches.

Two Evaluation Modes

| Aspect | CI/CD Evals | Production Monitoring | | ------ | ----------- | -------------------- | | When | Pre-deployment | Post-deployment, ongoing | | Data | Fixed dataset | Sampled traffic | | Goal | Prevent regression | Detect drift | | Response | Block deploy | Alert & analyze |

CI/CD Evaluations

from phoenix.client import Client

client = Client()

# Fast, deterministic checks
ci_evaluators = [
    has_required_format,
    no_pii_leak,
    safety_check,
    regression_test_suite,
]

# Small but representative dataset (~100 examples)
client.experiments.run_experiment(dataset=ci_dataset, task=task, evaluators=ci_evaluators)

Set thresholds: regression=0.95, safety=1.0, format=0.98.

Production Monitoring

Python

from phoenix.client import Client
from datetime import datetime, timedelta

client = Client()

# Sample recent traces (last hour)
traces = client.traces.get_traces(
    project_identifier="my-app",
    start_time=datetime.now() - timedelta(hours=1),
    include_spans=True,
    limit=100,
)

# Run evaluators on sampled traffic
for trace in traces:
    results = run_evaluators_async(trace, production_evaluators)
    if any(r["score"] < 0.5 for r in results):
        alert_on_failure(trace, results)

TypeScript

import { getTraces } from "@arizeai/phoenix-client/traces";
import { getSpans } from "@arizeai/phoenix-client/spans";

// Sample recent traces (last hour)
const { traces } = await getTraces({
  project: { projectName: "my-app" },
  startTime: new Date(Date.now() - 60 * 60 * 1000),
  includeSpans: true,
  limit: 100,
});

// Or sample spans directly for evaluation
const { spans } = await getSpans({
  project: { projectName: "my-app" },
  startTime: new Date(Date.now() - 60 * 60 * 1000),
  limit: 100,
});

// Run evaluators on sampled traffic
for (const span of spans) {
  const results = await runEvaluators(span, productionEvaluators);
  if (results.some((r) => r.score < 0.5)) {
    await alertOnFailure(span, results);
  }
}

Prioritize: errors → negative feedback → random sample.

Feedback Loop

Production finds failure → Error analysis → Add to CI dataset → Prevents future regression