AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

phoenix-evals

Build and run evaluators for AI/LLM applications using Phoenix.

Original source
github
License
MIT
Pinned revision
5269f7ea2d6c3a9bfda5ed878b6726cf509121fd

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read github--phoenix-evals at revision 5269f7ea2d6c3a9bfda5ed878b6726cf509121fd, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 35

references/experiments-synthetic-python.md

Experiments: Generating Synthetic Test Data

Creating diverse, targeted test data for evaluation.

Dimension-Based Approach

Define axes of variation, then generate combinations:

dimensions = {
    "issue_type": ["billing", "technical", "shipping"],
    "customer_mood": ["frustrated", "neutral", "happy"],
    "complexity": ["simple", "moderate", "complex"],
}

Two-Step Generation

  1. Generate tuples (combinations of dimension values)
  2. Convert to natural queries (separate LLM call per tuple)
# Step 1: Create tuples
tuples = [
    ("billing", "frustrated", "complex"),
    ("shipping", "neutral", "simple"),
]

# Step 2: Convert to natural query
def tuple_to_query(t):
    prompt = f"""Generate a realistic customer message:
    Issue: {t[0]}, Mood: {t[1]}, Complexity: {t[2]}
    
    Write naturally, include typos if appropriate. Don't be formulaic."""
    return llm(prompt)

Target Failure Modes

Dimensions should target known failures from error analysis:

# From error analysis findings
dimensions = {
    "timezone": ["EST", "PST", "UTC", "ambiguous"],  # Known failure
    "date_format": ["ISO", "US", "EU", "relative"],   # Known failure
}

Quality Control

  • Validate: Check for placeholder text, minimum length
  • Deduplicate: Remove near-duplicate queries using embeddings
  • Balance: Ensure coverage across dimension values

When to Use

| Use Synthetic | Use Real Data | | ------------- | ------------- | | Limited production data | Sufficient traces | | Testing edge cases | Validating actual behavior | | Pre-launch evals | Post-launch monitoring |

Sample Sizes

| Purpose | Size | | ------- | ---- | | Initial exploration | 50-100 | | Comprehensive eval | 100-500 | | Per-dimension | 10-20 per combination |