AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

diagnosing-experiment-health

Runs diagnostics and a health check on one PostHog experiment: its setup, exposures, results and changes during the run. Covers 0 exposures, sample ratio mismatch, lost or uneven exposures, users in multiple variants, identity faults, who the experiment counts, significance traps (peeking, A/A, 'target reached', Bayesian vs Frequentist, sequential, CUPED), mid-run edits, pause and freeze, and survey follow-up. TRIGGER when: user asks 'is my experiment healthy / set up right / biased?' or 'why 0 exposures?', asks why it counts different people than their insight, mentions the bias or mismatch warning, an uneven split, significance that flips, numbers that disagree with their SQL, results that changed after an edit, or wants user feedback on an experiment. DO NOT TRIGGER when: creating an experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle).

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--diagnosing-experiment-health at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 8

references/empty-experiment.md · 24000

EDIUM]

Eligibility filtering should happen before you call the flag — otherwise unaffected users are pulled into the analysis and the picture gets noisy. This shows up as exposures being much higher than expected and metric rates unexpectedly low.

The same shape has other sources: a flag that is read on more surfaces than the one under test, and an experiment on new users whose flag existing users also evaluate. In each, the experiment counts people the decision is not about (D12 in numbers-vs-sql.md).

Fix: structure the code as: eligibility check → flag check → render. Not: flag check → eligibility → render. Where the code cannot change, narrow who counts: a filter in the exposure criteria, which removes a person only when every exposure event of that person fails it (B12 lists what such a filter can read), or a custom exposure event sent at the point of the change.

B10 — "Variant always undefined / false" [MEDIUM]

Almost always one of:

  • B1 (wrong evaluation method), when the code reads a payload: getFeatureFlagPayload() returns the payload, which is empty for a flag without one
  • B2 (identify() timing), on a flag whose release conditions read person properties that are set at identify(): a read before it gets no variant
  • posthog is not defined (SDK init order — initialize PostHog before any flag call)
  • The code reads the flag before the SDK has loaded the flags. posthog-js then returns undefined and sends no $feature_flag_called, unless the browser holds flags from an earlier visit
  • The flag is genuinely off — feature_flag.active === false (status is paused), or rollout 0%, or the user is outside release conditions
  • The flag buckets on the device id, and the call carries none (A10 in bias-and-skew.md)
  • The code reads the multivariate flag as a boolean and gets true, or replaces "no variant" with a default of its own

Fix: walk the user through their SDK setup. Verify in this order: (a) is PostHog initialized? (b) is the flag active and rolled out? (c) is the right variant key being requested?

When only a part of the audience gets no variant, the question is who: A10 in bias-and-skew.md.

B11 — $feature/<key> is missing on metric events [HIGH]

For a current experiment this is not a fault. Only the exposure event decides a person's variant. Metric events are matched to that person and need no flag property, for every metric type. A user who adds $feature/<flag-key> to metric events changes nothing in the results. A legacy experiment shows no results at all (E13 in mid-run-changes.md).

Where the property does matter:

  • A custom exposure event needs it (B4).
  • The user's own insight or SQL that filters or breaks down by $feature/<flag-key>. It undercounts events from a server SDK, which adds the property only when the capture call is given the evaluated flags or asked to evaluate them [MEDIUM]. It also overcounts in the other direction: the property is on events of everyone whose SDK holds the flag's value, including people who were never exposed, so it is not a filter for participants.

Verify directly. A missing conversion has another cause: B2 (ids), B8 (order), D1 in numbers-vs-sql.md (scope).

Part of the public docs still asks for the property on metric events. The exposures page states the current rule.

B12 — An exposure criteria filter that cannot match [MEDIUM]

Symptom (in the user's vocabulary): "exposures dropped to zero after I added a filter", "the experiment shows far fewer users than the flag", "I copied the experiment and the new one is empty".

Behavior. A filter in the exposure criteria is evaluated on each exposure event. It reads the event's own properties, and person properties either as they were attached to that event or as the person holds them now: which of the two depends on a project setting that the tools do not show. A filter that names something the exposure event cannot carry removes every exposure, and nothing reports it.

Shapes:

  • A property the exposure event does not have. A browser property (device type, URL) on an exposure that a server sends.
  • A person property that is set after the exposure, such as a sign-up timestamp on an exposure that happens before sign-up. This shape occurs only where person properties are read as they were attached to the event.
  • A misspelled property name or value.
  • Criteria copied from another experiment that name the other experiment's flag or event.
  • An exposure event the project has never received.
  • A URL filter, or another event-property filter, that is meant to keep people out. It drops the events that do not match. A person with one matching exposure event is in, with everything they did.

Evidence to gather.

  • exposure_criteria.exposure_config from experiment-get: the event and its properties.
  • The raw-responses query once without and once with the filter's condition. A count that falls to zero, or nearly, names the filter. One condition at a time when there are several.
  • For a person-property filter: the raw-responses query with the same condition on person.properties.<key>. That query reads person properties the same way the experiment does, so the check holds whichever way the project reads them.

Co-occurs with: B7, and D9 in numbers-vs-sql.md.

Recommendation. Remove or correct the filter. Filter on what the exposure event carries. To restrict the experiment to a group of people, use the flag's release conditions. An edit of the exposure criteria recomputes the whole run (E5 in mid-run-changes.md).

If none of the above: the code path may not be running

Two sub-cases:

Never had exposures (the experiment has shown 0 since launch). After B1–B12, check the obvious:

  • Has the deploy with the flag-reading code shipped to production?
  • Is real traffic flowing through that code path?
  • Does the code read the experiment's flag key, and not another key?
  • Is the date range correct (start_date in the future, etc.)?

Ask explicitly. The "empty experiment" shape often resolves to a feature flag still on a feature branch that hasn't merged, or a page that calls the flag not being live yet.

Exposures were healthy then stopped (the experiment ran for weeks/months, then the exposure count plateaued and never moved again). Before investigating, check the experiment's status from Step 1:

  • exposure_frozen — someone froze exposure. The plateau is the intended behavior: enrollment is closed, metrics still flow (E16 in mid-run-changes.md).
  • paused — the flag is off.
  • stopped — the results end at the end date.

Then check where the project keeps its flag calls: project-get returns flag_evaluations_mode. At 2, $feature_flag_called no longer arrives in events for the projects that setting covers, and an experiment that counts that event gains no exposures, without a warning. That is no fault of the SDK or of the application: say so, and point the user to PostHog support.

Then read the flag's change history, before any conclusion about the application:

  • A change of the flag after launch can explain the plateau by itself. A split edited to 0% for one variant, a variant pinned on a broad release condition, or a rollout cut to 0 stops one arm or all of them while status still reads running (A5, A6 and A7b in bias-and-skew.md, E3 in mid-run-changes.md).
  • No change of the flag after launch means that an edit of the rollout, the split or the release conditions cannot explain it. What is left: the application stopped calling the flag, a release condition on a cohort or a property stopped matching new people, the audience has no new people, or the project's test-account filter changed, which the experiment's history does not show.

Verify directly:

  • exposures.timeseries in the exposure data: the cumulative count per variant stops growing. One arm flat while the other grows points to the flag or to one code path. All arms flat points to the application, or to an audience without new people: the series counts each person once.
  • last_seen per variant in the raw-responses query, over the last days: no recent events means the application stopped sending them, or that only people who already sent the event still reach the code (B3). Both variants silent is a code-path removal, or an audience without new people. One variant silent while the other still fires is a one-sided change, in the flag or in the code.

Common causes:

  • The flag-reading call was removed in a refactor (most common).
  • The page or component that hosts the flag-read was deprecated or rerouted (e.g. URL restructuring moved the eligible traffic onto a different page that doesn't read this flag).
  • A different flag is now serving the same UX (intentional migration that wasn't paired with ending the original experiment).

Recommend:

  • If the hypothesis is settled enough: end the experiment with the appropriate conclusion (won / lost / inconclusive). The metric data accumulated before the plateau is the experiment's documented outcome. Don't ship the variant unless the code path is being restored — an "end + ship" on a dormant flag flips the variant distribution to a UX that isn't being served anyway.
  • If you want to keep running the hypothesis: restore the flag-reading call in the application code, then either continue (and treat the pre-/post-resumption windows separately) or reset + relaunch for a clean comparison window.

SDK-side fallback. If B1, B2, and B10 are all on the table and you can't pin one down, invoke the posthog:diagnosing-sdk-health skill — an outdated SDK is one possible cause of the "no exposures at all" shape (missing instrumentation, broken identify() ordering, deprecated flag methods).