AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

creating-online-evaluations

Author continuously-running online evaluations in PostHog AI observability, grounded in real failure modes you've identified. Use when the user wants evaluations that automatically score new generations or whole traces going forward — "create an eval to catch X", "continuously check that responses do Y", "turn these failures into evals". Covers letting the explored data decide how many evals to create, proposing that set for the user to pick, choosing the target and eval type (hog / llm_judge / sentiment), configuring a provider and model for an llm_judge eval (a provider key gates enabling, not creation), scoping which generations trigger it via conditions, creating disabled, verifying scope, enabling, and backfilling past data. Proposes a sentiment eval when no failure mode is worth catching. Finding and ranking the failure modes worth evaluating is its own job — use exploring-ai-failures first. To debug or manage evaluations that already exist, use exploring-llm-evaluations.

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--creating-online-evaluations at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 2

SKILL.md · 24000

dow, conditions, and rerun_existing the user wants. Tell the user total_units, the returned window (it is clamped, so it can differ from the one you asked for), and what each unit costs: one run of the eval, counted as an AI observability event, plus a model call for an llm_judge. already_evaluated_units already have a result and are left out unless rerun_existing is true. 2. Call posthog:llma-evaluation-backfill-create only after the user says yes. Pass the window_start and window_end the estimate returned, not the ones you asked for, with the same conditions and rerun_existing, so the run matches what they approved. 3. Track it with posthog:llma-evaluation-backfill-get. Skipped units were already being evaluated by the eval itself, so they are covered, not missed. Once it completes, a remaining_count above zero means units were left without a result; offer another backfill over the same range. Offer it once. If the second run leaves units behind again, the eval keeps failing on them, so tell the user instead of offering a third. 4. If posthog:llma-evaluation-backfill-create says it could not confirm the start, call posthog:llma-evaluation-backfill-list before trying again, because the backfill may be running. 5. To stop a backfill, call posthog:llma-evaluation-backfill-cancel. Evaluations it already started still finish.

One backfill runs per evaluation at a time.

Scoping with conditions

conditions is a list of condition sets — OR between sets, AND within a set's properties. Each set is { id, rollout_percentage, properties[] }. There is no time window inside conditions; sampling is only rollout_percentage (0–100). Property filters use the standard PostHog shape (key, type, operator, value). For trace targets, these filters still select the generation that triggers the eventual whole-trace evaluation.

"conditions": [
  { "id": "openai",    "rollout_percentage": 100, "properties": [{"key": "$ai_provider", "type": "event", "operator": "exact", "value": "openai"}] },
  { "id": "anthropic", "rollout_percentage": 25,  "properties": [{"key": "$ai_provider", "type": "event", "operator": "exact", "value": "anthropic"}] }
]

Constructing UI links

Build links with posthog:generate-app-url — never hand-write the host or the /project/<id>/ prefix. The url must be a canonical catalog template; pass concrete ids via params, never inline them into the path.

  • Evaluations list: generate-app-url {url: "/ai-evals/evaluations"}
  • Single evaluation: generate-app-url {url: "/ai-evals/evaluations/{id}", params: {id: "<evaluation_id>"}}

These resolve to the correct region host and project prefix (e.g. https://us.posthog.com/project/<id>/ai-evals/evaluations/<evaluation_id>). Surface the link after creating so the user can review and toggle it in the UI.

Tips

  • Evals come from real failures, not generic metrics. Start from a failure found in this product's traffic (via exploring-ai-failures), not from "let's measure hallucination". A metric nobody traced back to a real bad output is noise.
  • One eval, one failure mode — and as many evals as the data justifies. Different failure modes need different evals; don't make one eval try to catch everything, and don't default to creating exactly one when the traces showed several modes worth watching. Related modes that one check tells apart are the exception: they fit one categorical eval.
  • Propose, then create what they picked. Show the candidate set in plain language, a line or two each, and wait for the user to choose. Long per-eval write-ups get skimmed, not read.
  • Nothing found still has an answer. Traces read but no failure mode worth an eval means proposing a sentiment eval, not returning empty-handed. No $ai_generation events at all is the exception — no eval can fire, so send them to instrument-llm-analytics instead.
  • Suggest changes along with the eval if possible. If it's clear a prompt change would fix the issue, for instance, set up the eval but also suggest to the user they change the prompt: they should soon see the eval go from low pass rate to a higher pass rate.
  • hog first. No provider key, no AI approval, deterministic. Reach for llm_judge only when the criterion genuinely can't be coded.
  • Always create disabled, verify scope, then enable. An eval firing on the wrong events is worse than none — noise, and (for llm_judge) cost.
  • A judge draft needs only a provider and model. A usable provider key is needed to enable it, not to save it, so no ok key means keep it disabled — not stop. provider_key_id may be null to run on the team's active key.
  • bytecode is server-written for hog evals — never pass it; send only evaluation_config.source.
  • For cluster-scoped evals, identify the cluster with exploring-llm-clusters, then translate its event filter into conditions.

Related skills

  • exploring-ai-failures — find and rank the failure modes worth evaluating — do this first
  • exploring-llm-evaluations — debug and manage evaluations that already exist
  • exploring-llm-clusters — identify a cluster to scope cluster-targeted eval conditions