AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

phoenix-evals

Build and run evaluators for AI/LLM applications using Phoenix.

Original source
github
License
MIT
Pinned revision
5269f7ea2d6c3a9bfda5ed878b6726cf509121fd

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read github--phoenix-evals at revision 5269f7ea2d6c3a9bfda5ed878b6726cf509121fd, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 35

references/evaluators-overview.md

Evaluators: Overview

When and how to build automated evaluators.

Decision Framework

Should I Build an Evaluator?
        │
        ▼
Can I fix it with a prompt change?
    YES → Fix the prompt first
    NO  → Is this a recurring issue?
          YES → Build evaluator
          NO  → Add to watchlist

Don't automate prematurely. Many issues are simple prompt fixes.

Evaluator Requirements

  1. Clear criteria - Specific, not "Is it good?"
  2. Labeled test set - 100+ examples with human labels
  3. Measured accuracy - Know TPR/TNR before deploying

Evaluator Lifecycle

  1. Discover - Error analysis reveals pattern
  2. Design - Define criteria and test cases
  3. Implement - Build code or LLM evaluator
  4. Calibrate - Validate against human labels
  5. Deploy - Add to experiment/CI pipeline
  6. Monitor - Track accuracy over time
  7. Maintain - Update as product evolves

What NOT to Automate

  • Rare issues - <5 instances? Watchlist, don't build
  • Quick fixes - Fixable by prompt change? Fix it
  • Evolving criteria - Stabilize definition first