SKILL.md · 24000
dow, conditions, and rerun_existing the
user wants. Tell the user total_units, the
returned window (it is clamped, so it can differ from the one you asked for), and what each unit costs:
one run of the eval, counted as an AI observability event, plus a model call for an llm_judge.
already_evaluated_units already have a result and are left out unless rerun_existing is true.
2. Call posthog:llma-evaluation-backfill-create only after the user says yes. Pass the window_start and
window_end the estimate returned, not the ones you asked for, with the same conditions and
rerun_existing, so the run matches what they approved.
3. Track it with posthog:llma-evaluation-backfill-get. Skipped units were already being evaluated by the
eval itself, so they are covered, not missed. Once it completes, a remaining_count above zero means
units were left without a result; offer another backfill over the same range. Offer it once. If the
second run leaves units behind again, the eval keeps failing on them, so tell the user instead of
offering a third.
4. If posthog:llma-evaluation-backfill-create says it could not confirm the start, call
posthog:llma-evaluation-backfill-list before trying again, because the backfill may be running.
5. To stop a backfill, call posthog:llma-evaluation-backfill-cancel. Evaluations it already started
still finish.
One backfill runs per evaluation at a time.
Scoping with conditions
conditions is a list of condition sets — OR between sets, AND within a set's properties. Each
set is { id, rollout_percentage, properties[] }. There is no time window inside conditions; sampling is
only rollout_percentage (0–100). Property filters use the standard PostHog shape
(key, type, operator, value). For trace targets, these filters still select the generation that
triggers the eventual whole-trace evaluation.
"conditions": [
{ "id": "openai", "rollout_percentage": 100, "properties": [{"key": "$ai_provider", "type": "event", "operator": "exact", "value": "openai"}] },
{ "id": "anthropic", "rollout_percentage": 25, "properties": [{"key": "$ai_provider", "type": "event", "operator": "exact", "value": "anthropic"}] }
]
Constructing UI links
Build links with posthog:generate-app-url — never hand-write the host or the /project/<id>/ prefix.
The url must be a canonical catalog template; pass concrete ids via params, never inline them into the path.
- Evaluations list:
generate-app-url {url: "/ai-evals/evaluations"} - Single evaluation:
generate-app-url {url: "/ai-evals/evaluations/{id}", params: {id: "<evaluation_id>"}}
These resolve to the correct region host and project prefix (e.g.
https://us.posthog.com/project/<id>/ai-evals/evaluations/<evaluation_id>). Surface the link after
creating so the user can review and toggle it in the UI.
Tips
- Evals come from real failures, not generic metrics. Start from a failure found in this product's
traffic (via
exploring-ai-failures), not from "let's measure hallucination". A metric nobody traced back to a real bad output is noise. - One eval, one failure mode — and as many evals as the data justifies. Different failure modes need different evals; don't make one eval try to catch everything, and don't default to creating exactly one when the traces showed several modes worth watching. Related modes that one check tells apart are the exception: they fit one categorical eval.
- Propose, then create what they picked. Show the candidate set in plain language, a line or two each, and wait for the user to choose. Long per-eval write-ups get skimmed, not read.
- Nothing found still has an answer. Traces read but no failure mode worth an eval means proposing a
sentimenteval, not returning empty-handed. No$ai_generationevents at all is the exception — no eval can fire, so send them toinstrument-llm-analyticsinstead. - Suggest changes along with the eval if possible. If it's clear a prompt change would fix the issue, for instance, set up the eval but also suggest to the user they change the prompt: they should soon see the eval go from low pass rate to a higher pass rate.
hogfirst. No provider key, no AI approval, deterministic. Reach forllm_judgeonly when the criterion genuinely can't be coded.- Always create disabled, verify scope, then enable. An eval firing on the wrong events is worse than none — noise, and (for llm_judge) cost.
- A judge draft needs only a provider and model. A usable provider key is needed to enable it, not to
save it, so no
okkey means keep it disabled — not stop.provider_key_idmay benullto run on the team's active key. bytecodeis server-written for hog evals — never pass it; send onlyevaluation_config.source.- For cluster-scoped evals, identify the cluster with
exploring-llm-clusters, then translate its event filter intoconditions.
Related skills
exploring-ai-failures— find and rank the failure modes worth evaluating — do this firstexploring-llm-evaluations— debug and manage evaluations that already existexploring-llm-clusters— identify a cluster to scope cluster-targeted eval conditions