AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

arize-evaluator

Handles LLM-as-judge evaluation workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, continuous monitoring, or improve evaluator prompt.

Original source
github
License
MIT
Pinned revision
5269f7ea2d6c3a9bfda5ed878b6726cf509121fd

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read github--arize-evaluator at revision 5269f7ea2d6c3a9bfda5ed878b6726cf509121fd, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 3

SKILL.md · 24000

ings`** is where you wire it to the actual fields in a specific project or experiment. This lets the same evaluator run across multiple projects and experiments without modification.

2. Default to binary labels

Use exactly two clear string labels (e.g. hallucinated / factual, correct / incorrect, pass / fail). Binary labels are:

  • Easiest for the judge model to produce consistently
  • Most common in the industry
  • Simplest to interpret in dashboards

If the user insists on more than two choices, that's fine — but recommend binary first and explain the tradeoff (more labels → more ambiguity → lower inter-rater reliability).

3. Be explicit about what the model must return

The template must tell the judge model to respond with only the label string — nothing else. The label strings in the prompt must exactly match the labels in --classification-choices (same spelling, same casing).

Good:

Respond with exactly one of these labels: hallucinated, factual

Bad (too open-ended):

Is this hallucinated? Answer yes or no.

4. Keep temperature low

Pass --invocation-params '{"temperature": 0}' for reproducible scoring. Higher temperatures introduce noise into evaluation results.

5. Use --include-explanations for debugging

During initial setup, always include explanations so you can verify the judge is reasoning correctly before trusting the labels at scale.

6. Pass the template in single quotes in bash

Single quotes prevent the shell from interpolating {variable} placeholders. Double quotes will cause issues:

# Correct
--template 'Judge this: {input} → {output}'

# Wrong — shell may interpret { } or fail
--template "Judge this: {input} → {output}"

7. Always set --classification-choices to match your template labels

The labels in --classification-choices must exactly match the labels referenced in --template (same spelling, same casing). Omitting --classification-choices causes task runs to fail with "missing rails and classification choices."


Troubleshooting

| Problem | Solution | |---------|----------| | ax: command not found | See references/ax-setup.md | | 401 Unauthorized | API key may not have access to this space. Verify at https://app.arize.com/admin > API Keys | | Evaluator not found | ax evaluators list --space SPACE | | Integration not found | ax ai-integrations list --space SPACE | | Task not found | ax tasks list --space SPACE | | project and dataset-id are mutually exclusive | Use only one when creating a task | | experiment-ids required for dataset tasks | Add --experiment-ids to create and trigger-run | | sampling-rate only valid for project tasks | Remove --sampling-rate from dataset tasks | | Validation error on ax spans export | Project name usually works; if you still get a validation error, look up the base64 project ID via ax projects list --space SPACE -o json and use the id field instead | | Template validation errors | Use single-quoted --template '...' in bash; single braces {var}, not double {{var}} | | Run stuck in pending | ax tasks get-run RUN_ID; then ax tasks cancel-run RUN_ID | | Run cancelled ~1s | Integration credentials invalid — check AI integration | | Run cancelled ~3min | Found spans but LLM call failed — wrong model name or bad key | | Run completed, 0 spans | Widen time window; eval index may not cover older data | | No scores in UI | Fix column_mappings to match real paths on your spans/runs | | Scores look wrong | Add --include-explanations and inspect judge reasoning on a few samples | | Evaluator cancels on wrong span kind | Match query_filter and column_mappings to LLM vs CHAIN spans | | Time format error on trigger-run | Use 2026-03-21T09:00:00 — no trailing Z | | Run failed: "missing rails and classification choices" | Add --classification-choices '{"label_a": 1, "label_b": 0}' to ax evaluators create — labels must match the template | | Run completed, all spans skipped | Query filter matched spans but column mappings are wrong or template variables don't resolve — export a sample span and verify paths | | query_filter set but 0 spans scored | The filter attribute may not be indexed in the eval index. attributes.metadata.* and custom attributes are often not indexed. Use span_kind or attributes.llm.model_name instead, or remove the filter to confirm spans exist in the window. |

Diagnosing cancelled runs

When a task run is cancelled (status cancelled), follow this checklist in order:

1. Check integration credentials

ax ai-integrations list --space SPACE -o json

Verify the integration ID used by the evaluator exists and has valid credentials. If the integration was deleted or the API key expired, the run cancels within ~1 second.

2. Verify the model name

ax evaluators get EVALUATOR_NAME --space SPACE -o json

Check the model_name field. A typo or deprecated model causes the LLM call to fail and the run to cancel after ~3 minutes.

3. Export a sample span/run and compare paths to column_mappings

For project tasks:

ax spans export PROJECT --space SPACE -l 1 --days 7 --stdout | python3 -m json.tool

For experiment tasks:

ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE --stdout | python3 -c "import sys,json; runs=json.load(sys.stdin); print(json.dumps(runs[0], indent=2)) if runs else print('No runs')"

Compare the exported JSON paths against the task's column_mappings. For each template variable, confirm the mapped path actually exists. Common mismatches:

  • Mapping output to attributes.output.value on an experiment run (should be just output)
  • Mapping input to attributes.input.value on a CHAIN span when the actual path is attributes.llm.input_messages
  • Mapping context to a path that doesn't exist on the span kind being filtered

4. Check that data_start_time is not epoch

If trigger-run used a start time of 0, 1970-01-01, or an empty string, the time window is invalid. Always derive from real span timestamps:

ax spans export PROJECT --space SPACE -l 5 --days 30 --stdout | python3 -c "
import sys, json
spans = json.load(sys.stdin)
for s in spans:
    print(s.get('start_time', 'N/A'), s.get('end_time', 'N/A'))
"

5. Verify span kind matches evaluator scope

If the evaluator was created with --data-granularity trace but the task's query_filter is span_kind = 'LLM', the run may find no qualifying data and cancel. Ensure the granularity and filter are consistent.

6. Check that all template variables resolve

Every {variable} in the evaluator template must have a corresponding column_mappings entry that resolves to a non-null value. Test resolution against a real span:

ax spans export PROJECT --space SPACE -l 3 --days 7 --stdout | python3 -c "
import sys, json
spans = json.load(sys.stdin)
# Replace these paths with your actual column_mappings values
mappings = {'input': 'attributes.input.value', 'output': 'attributes.output.value'}
for i, span in enumerate(spans):
    print(f'--- Span {i} ---')
    for var, path in mappings.items():
        parts = path.split('.')
        val = span
        for p in parts:
            val = val.get(p) if isinstance(val, dict) else None
        status = 'FOUND' if val else 'MISSING'
        print(f'  {var} ({path}): {status} — {str(val)[:80] if val else \"null\"}')
"

If any variable shows MISSING on all spans, fix the column mapping or adjust query_filter to target a different span kind.


Related Skills

  • arize-ai-provider-integration: Full CRUD for LLM provider integrations (create, update, delete credentials)
  • arize-trace: Export spans to discover column paths and time ranges
  • arize-experiment: Create experiments and export runs for experiment column mappings
  • arize-dataset: Export dataset examples to find input fields when runs omit them
  • arize-link: Deep links to evaluators and tasks in the Arize UI

Save Credentials for Future Use

See references/ax-profiles.md § Save Credentials for Future Use.