AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

phoenix-evals

Build and run evaluators for AI/LLM applications using Phoenix.

Original source
github
License
MIT
Pinned revision
5269f7ea2d6c3a9bfda5ed878b6726cf509121fd

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read github--phoenix-evals at revision 5269f7ea2d6c3a9bfda5ed878b6726cf509121fd, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 35

references/evaluators-rag.md

Evaluators: RAG Systems

RAG has two distinct components requiring different evaluation approaches.

Two-Phase Evaluation

RETRIEVAL                    GENERATION
─────────                    ──────────
Query → Retriever → Docs     Docs + Query → LLM → Answer
         │                              │
    IR Metrics              LLM Judges / Code Checks

Debug retrieval first using IR metrics, then tackle generation quality.

Retrieval Evaluation (IR Metrics)

Use traditional information retrieval metrics:

| Metric | What It Measures | | ------ | ---------------- | | Recall@k | Of all relevant docs, how many in top k? | | Precision@k | Of k retrieved docs, how many relevant? | | MRR | How high is first relevant doc? | | NDCG | Quality weighted by position |

# Requires query-document relevance labels
def recall_at_k(retrieved_ids, relevant_ids, k=5):
    retrieved_set = set(retrieved_ids[:k])
    relevant_set = set(relevant_ids)
    if not relevant_set:
        return 0.0
    return len(retrieved_set & relevant_set) / len(relevant_set)

Creating Retrieval Test Data

Generate query-document pairs synthetically:

# Reverse process: document → questions that document answers
def generate_retrieval_test(documents):
    test_pairs = []
    for doc in documents:
        # Extract facts, generate questions
        questions = llm(f"Generate 3 questions this document answers:\n{doc}")
        for q in questions:
            test_pairs.append({"query": q, "relevant_doc_id": doc.id})
    return test_pairs

Generation Evaluation

Use LLM judges for qualities code can't measure:

| Eval | Question | | ---- | -------- | | Faithfulness | Are all claims supported by retrieved context? | | Relevance | Does answer address the question? | | Completeness | Does answer cover key points from context? |

from phoenix.evals import ClassificationEvaluator, LLM

FAITHFULNESS_TEMPLATE = """Given the context and answer, is every claim in the answer supported by the context?

<context>{{context}}</context>
<answer>{{output}}</answer>

"faithful" = ALL claims supported by context
"unfaithful" = ANY claim NOT in context

Answer (faithful/unfaithful):"""

faithfulness = ClassificationEvaluator(
    name="faithfulness",
    prompt_template=FAITHFULNESS_TEMPLATE,
    llm=LLM(provider="openai", model="gpt-4o"),
    choices={"unfaithful": 0, "faithful": 1}
)

RAG Failure Taxonomy

Common failure modes to evaluate:

retrieval_failures:
  - no_relevant_docs: Query returns unrelated content
  - partial_retrieval: Some relevant docs missed
  - wrong_chunk: Right doc, wrong section

generation_failures:
  - hallucination: Claims not in retrieved context
  - ignored_context: Answer doesn't use retrieved docs
  - incomplete: Missing key information from context
  - wrong_synthesis: Misinterprets or miscombines sources

Evaluation Order

  1. Retrieval first - If wrong docs, generation will fail
  2. Faithfulness - Is answer grounded in context?
  3. Answer quality - Does answer address the question?

Fix retrieval problems before debugging generation.