AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

phoenix-evals

Build and run evaluators for AI/LLM applications using Phoenix.

Original source
github
License
MIT
Pinned revision
5269f7ea2d6c3a9bfda5ed878b6726cf509121fd

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read github--phoenix-evals at revision 5269f7ea2d6c3a9bfda5ed878b6726cf509121fd, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 35

references/production-guardrails.md

Production: Guardrails vs Evaluators

Guardrails block in real-time. Evaluators measure asynchronously.

Key Distinction

Request → [INPUT GUARDRAIL] → LLM → [OUTPUT GUARDRAIL] → Response
                                            │
                                            └──→ ASYNC EVALUATOR (background)

Guardrails

| Aspect | Requirement | | ------ | ----------- | | Timing | Synchronous, blocking | | Latency | < 100ms | | Purpose | Prevent harm | | Type | Code-based (deterministic) |

Use for: PII detection, prompt injection, profanity, length limits, format validation.

Evaluators

| Aspect | Characteristic | | ------ | -------------- | | Timing | Async, background | | Latency | Can be seconds | | Purpose | Measure quality | | Type | Can use LLMs |

Use for: Helpfulness, faithfulness, tone, completeness, citation accuracy.

Decision

| Question | Answer | | -------- | ------ | | Must block harmful content? | Guardrail | | Measuring quality? | Evaluator | | Need LLM judgment? | Evaluator | | < 100ms required? | Guardrail | | False positives = angry users? | Evaluator |

LLM Guardrails: Rarely

Only use LLM guardrails if:

  • Latency budget > 1s
  • Error cost >> LLM cost
  • Low volume
  • Fallback exists

Key Principle: Guardrails prevent harm (block). Evaluators measure quality (log).