AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

diagnosing-experiment-health

Runs diagnostics and a health check on one PostHog experiment: its setup, exposures, results and changes during the run. Covers 0 exposures, sample ratio mismatch, lost or uneven exposures, users in multiple variants, identity faults, who the experiment counts, significance traps (peeking, A/A, 'target reached', Bayesian vs Frequentist, sequential, CUPED), mid-run edits, pause and freeze, and survey follow-up. TRIGGER when: user asks 'is my experiment healthy / set up right / biased?' or 'why 0 exposures?', asks why it counts different people than their insight, mentions the bias or mismatch warning, an uneven split, significance that flips, numbers that disagree with their SQL, results that changed after an edit, or wants user feedback on an experiment. DO NOT TRIGGER when: creating an experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle).

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--diagnosing-experiment-health at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 8

references/numbers-vs-sql.md · 24000

).

  • Late or backfilled events. A project with precomputed results keeps each day's exposures and metric events once that day is a few days old. Events that arrive later for that day stay out of the results for about two months. is_precomputed: true on a metric's stored result says that the project precomputes and that this metric's exposures were read from that store.
  • A removed variant. Results computed before a variant left the flag keep that variant until the next calculation: up to 24 hours for a cached result.
  • A changed setting. An edit of a metric, of the exposure criteria or of a statistics setting on the page recalculates the whole run. Numbers that moved without new data are usually this: read experiment-activity. An edit through the API or a tool starts no calculation. The stored results of the changed metrics then come back empty until someone opens the page or a calculation runs.

Whatever the cause, a query_to far behind now on a running experiment, or far behind end_date on a stopped one, means that the stored numbers are old. Say how old before you interpret them.

A failed calculation is not a zero. A metric whose stored result has status: failed shows no value on the page (see "Metric rows with data: null" in diagnostic-snapshot.md).

Recommend: state query_to with every number you take from the stored results. When the user needs current numbers, the refresh control on the page recalculates, and so does experiment-metrics-recalculation-create: ask first, because it computes every metric. An ended experiment has no refresh control on the page: the tool is the way to compute up to the end date. On a stopped experiment that already has a calculation after the stop, neither brings in new data.

D9 — Applying a filter doesn't change the user count [MEDIUM]

Symptom: a filter is added to a metric (e.g. "by device = mobile") and the exposure / user count stays the same — only the conversion side moves. The conclusion looks like "the filter isn't working."

The experiment's denominator is the set of exposed users, fixed at exposure time. A filter on a property of the metric event acts as a gate within that fixed population — it changes who counts as converted, not who counts as in the experiment. The denominator correctly does not shrink. A retention metric with its own start event is the exception: its denominator is the exposed people who did the start event, so a filter on the start event shrinks it.

To shrink the denominator (i.e. only count users who match the filter as part of the experiment at all), encode the eligibility upstream — in release conditions, in a property filter on the exposure criteria (B12 in empty-experiment.md lists what such a filter can read), or with a custom exposure event that already filters.

Recommend: explain the scope difference. If the mental model comes from another A/B tool that subset-filters the population on metric properties, name the tool and explain the design choice explicitly.

D10 — No "current person properties" toggle on experiment metrics [MEDIUM]

Insights have a "Use person properties from query time" toggle (versus as-of-event). Experiment metrics do not expose this toggle. They read person properties in the project's person-properties mode: the value as of the time the event was captured when the project keeps person properties on events, the person's current value when it joins them at query time. Which of the two applies differs between projects.

This is intentional: the experiment's population needs to be stable across the run. If person properties were re-resolved at query time, the population a user falls into could change over the course of the experiment as their attributes change (plan upgrades, geo moves, etc.), which would invalidate the analysis.

A property that was set or backfilled after the events is therefore invisible to a breakdown or a filter when the project keeps person properties on events: the old events keep the old value. A person property synced from a data warehouse behaves the same way.

Recommend: for slices by "current state" attributes (e.g. "free vs paid as of today"), use one of:

  • A HogQL expression in the metric filter that reads the person's current properties (pdi.person.properties.<key>), accepting that the answer reflects the current state, not the state at exposure. Plain person.properties.<key> reads the value on the event when the project keeps person properties on events, and the current value otherwise. One filtered metric per slice replaces the breakdown.
  • A property captured on the exposure event (e.g. plan tier at the time of exposure), so the slice is stable and analysable as a breakdown.
  • A dynamic cohort for "currently paid" users in the flag's release conditions, before launch. Do not use a dynamic cohort in the exposure criteria to mirror a release condition: the two are evaluated at different times (A7 in bias-and-skew.md).

D11 — Metric definition traps [HIGH]

Metric configurations that produce numbers which look like broken data and are the definition doing exactly what it says. Each is readable from the metric's definition in experiment-get.

event: "" is not "all events". In an EventsNode, the event field is an equality filter against the event name. An empty string matches events literally named "" — i.e. none. The metric collapses to a constant per user (commonly 0). A mean or ratio metric left on the editor's "All events" placeholder is the same case: the editor stores the name All events, which matches no event, and the page rejects such an inline metric on save (next paragraph). A funnel step left on "All events" is the opposite: it is stored without an event name and counts every event. In a ratio metric the two halves can differ: an empty-string numerator matches nothing, and a denominator without an event counts every event.

An event the project has never received. A misspelled or not-yet-shipped event name gives a metric that is 0 in every variant. Inline metrics are checked against the project's events when they are saved, unless the caller passed allow_unknown_events. Shared metrics are not checked. Evidence: the event name is absent from the project's event definitions.

count(boolean_expression) counts non-null, not true. A HogQL math_hogql of the form count(properties.X = 'value') counts every event where the expression evaluates (i.e. every event where the property is set, true or false), not events where the expression is true. Use countIf(properties.X = 'value') for the "true" semantics, or sum(toInt(properties.X = 'value')) for an additive form.

An upper outlier bound on a rare event. With an upper percentile set and ignore_zeros off, the percentile is taken over all exposed people. When most people have a value of 0, the bound itself is 0 or close to it, and every value is clamped down to it: the metric shows no difference, or a difference that is an artifact of the clamp. Evidence: upper_bound_percentile on a mean metric (or on a ratio's numerator or denominator) with ignore_zeros false, and a metric event that few exposed people send. Fix: turn on ignore_zeros.

An event that only one variant sends [LOW]. When each arm reports its conversion with its own event and the metric names one of them, one arm cannot convert. The lift is enormous or undefined. Fix: one action that covers both events, used as the metric.

Validation signals from PostHog. A validation_failures entry of "baseline-mean-is-zero" is the system's tell that the baseline variant's values sum to 0 (for a ratio, its numerator). For a metric whose values cannot be negative, every person in the baseline variant contributed 0 to the metric — almost always one of the traps above. With it, no test variant gets an effect or an interval.

Verify directly. Read the metric's definition: source, series, numerator, denominator, the math and the outlier settings. The generated SQL is not part of the results an agent can read.

Recommend:

  • Replace event: "" with the actual event to measure.
  • For HogQL math: pick countIf(...) or sum(toInt(...)) over count(...) of a boolean.
  • Metric edits on a running experiment recompute the metric over the full duration (D8 says when). Flag this to the user before recommending so the post-edit numbers don't surprise them.

D12 — The experiment counts a wider population than the insight it is compared with [MEDIUM]

Symptom (in the user's vocabulary): "the experiment says flat, our dashboard shows a clear drop", "the conversion rate in the experiment is half of what the funnel shows", "most of the exposed users are existing customers".

Behavior. The experiment counts everyone who was exposed. The team's own funnel or dashboard often counts a narrower group: new visitors only, verified sign-ups only, one surface only. When the change targets that narrow group and the flag is evaluated for everyone, the people the decision is about are a small share of the exposed, and their effect is diluted or reversed in the experiment's number. The exposure settings look correct, because they are the defaults. The mismatch is between the exposed population and the population the decision is about.

Evidence to gather.

  • The reference. The insight the user trusts: its filters on persons and events (insight-get), and its numbers (insight-query). When the user names none, ask which funnel or dashboard the team decides by. Do not pick one for them.
  • The experiment. exposure_criteria and the flag's release conditions: is there a filter that keeps the same people out?
  • The difference, listed filter by filter. Typical: existing users in an acquisition experiment, unverified or bot sign-ups, a flag that is read on more pages than the one under test, a second experiment on the same surface (the finding-experiments skill lists the project's experiments by date and flag).

The reverse case, an insight that counts more than the experiment because nothing scopes it to exposed people, is D1.

MUST compare the exposed population with the population the decision is about before calling the setup sound.

MUST NOT conclude that exposure is fine because the exposure criteria are the defaults. The defaults say how exposures are counted, not who should be in the experiment.

Co-occurs with: B9 in empty-experiment.md, C12 in interpretation.md, D1.

Recommendation. Narrow the experiment to the people the decision is about: a release condition, a filter in the exposure criteria, or an exposure event at the point of the change. On a running experiment this is a mid-run change (E5 in mid-run-changes.md): an exposure-criteria filter recomputes the whole run (D8 says when) and moves no one between variants, a release condition changes who enrolls from now on. State the trade-off, and let the user choose between narrowing in place and a clean restart.

D13 — A metric's title does not match what it measures [MEDIUM]

Symptom (in the user's vocabulary): "the metric is called 7-day retention and the number makes no sense", "the sign-up metric counts something else", a result that contradicts the same-named insight.

Behavior [HIGH]. A metric's name is free text.

  • A typed name is never updated when the metric's events, math or windows change. It can describe an earlier definition.
  • A metric without a typed name shows a title derived from its events: the first event of a mean or a funnel, numerator and denominator of a ratio, start and completion event of a retention metric.
  • A shared metric shows the name of the shared metric. An edit of a shared metric changes it in every experiment that uses it, finished ones included.

Evidence to gather. For each metric the user reads: the name, then metric_type and the definition (source or series, numerator and denominator, start_event and completion_event, the conversion or retention window, the math). For a shared metric, also the experiment's change history: was the shared metric edited during the run?

MUST say what the metric measures now, from its definition, when the title and the definition disagree.

MUST NOT restate the title as the definition. A result explained by its title is wrong whenever the two differ.

Co-occurs with: D11, E6 and E10 in mid-run-changes.md.

Recommendation. Rename the metric to what it measures, or correct the definition. A definition change recomputes the metric over the whole run (D8 says when).