AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

diagnosing-experiment-health

Runs diagnostics and a health check on one PostHog experiment: its setup, exposures, results and changes during the run. Covers 0 exposures, sample ratio mismatch, lost or uneven exposures, users in multiple variants, identity faults, who the experiment counts, significance traps (peeking, A/A, 'target reached', Bayesian vs Frequentist, sequential, CUPED), mid-run edits, pause and freeze, and survey follow-up. TRIGGER when: user asks 'is my experiment healthy / set up right / biased?' or 'why 0 exposures?', asks why it counts different people than their insight, mentions the bias or mismatch warning, an uneven split, significance that flips, numbers that disagree with their SQL, results that changed after an edit, or wants user feedback on an experiment. DO NOT TRIGGER when: creating an experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle).

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--diagnosing-experiment-health at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 8

references/bias-and-skew.md · 24000

onditions compare against.

A5 — Flag/experiment state inconsistency [HIGH]

The experiment view shows one warning banner for a state in which the experiment and its flag disagree. Five states exist, checked from experiment-get alone:

| Banner caption | State | What the banner says to do | | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | "The experiment is paused" | Launched, no end date, the flag is disabled. status is paused | Resume or end the experiment | | "The experiment is running, but no new users are being exposed" | Running, every release condition has a rollout of 0% | End with a conclusion, or increase the rollout. People exposed earlier stay in the results | | "The experiment is running, but all users see a single variant" | Running, the first release condition matches everyone at 100%, and one variant holds 100% of the split | End with a conclusion, or adjust the variant distribution | | "The experiment is not running, but users are exposed to multiple variants" | Ended, the flag is active, more than one variant is above 0%, and a release condition is above 0% | Disable the flag, or resume the experiment | | "The experiment is not running, but users are exposed to multiple variants" | Not launched (also after a reset), not archived, the flag is active, more than one variant is above 0%, and a release condition is above 0% | Disable the flag, or start the experiment |

"Adjust the variant distribution" on a running experiment is a change of the split, with what that does to the people already exposed (E3 in mid-run-changes.md). Say so before the user restores a split.

What the banners do not cover:

  • A pinned variant. A release condition that forces one variant shows no banner (A7b).
  • A shipped variant. A ship ends the experiment, and the single-variant banner checks running experiments only (E7 in mid-run-changes.md).
  • Frozen exposure. status is exposure_frozen, and no banner appears (E16 in mid-run-changes.md).
  • A deleted flag.

A flag that was active before launch. Exposures before start_date are outside the analysis, and the hash is the same before and after launch, so they do not put people in more than one variant. The risk is the SDK's deduplication: a returning person who already sent the exposure event before launch may not send it again (B3 in empty-experiment.md).

Use managing-experiment-lifecycle for the correct lifecycle action.

A6 — Mid-run flag edits that rebucket already-exposed users [MEDIUM]

A person's variant follows from three things: the flag key, the identifier the flag buckets on, and the boundaries of the variant split. The variant key is not an input: a variant is a slot in the split, and the key is its label. The rollout of a release condition is drawn separately, so it decides whether a person is in, never which variant.

So flag edits after launch differ in what they do to a person who is already exposed:

| Edit after launch | Moves the person to another variant? | What the data shows | | ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | | The variant split changes. Also: one variant to 0%, or the variants reordered | Yes, everyone whose draw lies between an old and a new boundary. That includes people of variants the edit did not touch | The same person under a second variant key: $multiple under exclude. A 0% variant stops growing | | The bucketing identifier changes, or persistence across authentication is switched | Yes, for the people whose identifier changes. A persistence switch moves only those whose stored or anonymous id differs from their distinct_id. A new identifier is a new draw, and about half keep their variant by chance on a 50/50 split | $multiple rises from the day of the edit | | A release condition is added, tightened, or its rollout changes | No. People move in or out. Out means the flag returns false, which is no variant | More or fewer new exposures. No $multiple from this edit alone | | The same, when the condition a person now matches pins a variant | Yes, to the pinned variant | See A7 | | A variant key is renamed | No. The same slot gets a new label. The flag API rejects the rename while the experiment runs | Exposures under the old key leave the analysis, because only current keys count | | The flag's key is renamed | Yes. The key is an input of the draw, so everyone gets a new one. The flag API allows it while the experiment runs | Exposures sent before the rename carry the old key and leave the experiment |

Evidence. experiment-activity returns the changes of the experiment and of its linked flag, each with the values before and after. Look for a change of the flag's filters after start_date, and compare multivariate.variants, groups and the bucketing identifier on both sides. The same rows are in feature-flags-activity-retrieve, and in advanced-activity-logs-list when fields is left out or includes detail.changes. advanced-activity-logs-list is offered only on a plan with audit logs.

What an edit does not prove. A split change explains the people who were exposed on both sides of it. People in more than one variant who were exposed only before or only after the edit have another cause, usually A3. State the edit and its date. Do not attribute the whole $multiple share to it.

Recommend: treat a split change and an identifier change as contamination from the edit on. Reset and relaunch is the cleaner fix. Switching multiple_variant_handling to first_seen is the low-disruption mid-run option (per A1). A rollout change alone needs no repair (E1 and E2 in mid-run-changes.md).

A7 — Non-randomized assignment via release conditions [MEDIUM]

If the user is using release conditions to target specific cohorts to specific variants (e.g. iOS users see test, Android users see control), the resulting assignment is not random. PostHog's statistics assume randomization, so this invalidates the standard significance interpretation.

PostHog doesn't prevent this in the UI — but the user should understand that significance calculations are misleading in this setup. Nothing in the analysis leaves the pinned people out.

Verify directly. In experiment-get's response, scan feature_flag.filters.groups[] for any entry where variant is one of the flag's variant keys. That field is the per-release-group variant override: any user matching that group's properties[] is forced to that variant rather than being randomly bucketed. A variant: null (or missing field) means the group is randomized normally and A7 doesn't apply.

Three rules decide who is pinned:

  • Release conditions are evaluated in their stored order, and the first match wins. A pinned condition forces only the people who matched no earlier condition.
  • A variant that is not one of the flag's variant keys is ignored, and the person is bucketed normally.
  • A holdout is evaluated before every release condition. People in the holdout are not pinned.

When the override exists, also check whether the targeted cohort overlaps the project's test-account exclusion list (B7 in empty-experiment.md says where to read it). If the cohort is in the exclusion list and exposure_criteria.filterTestAccounts is not false, those users are filtered out of the analysis and the override is mostly a no-op for the metric (they were never going to count). If the cohort is not excluded (e.g. an external partner's email domain), the override contaminates the variant assignment for real users.

When the exclusion runs through a dynamic cohort, it lags [LOW]. The flag evaluates the person's properties live, and the analysis usually reads the cohort's stored membership, which is recalculated on a schedule. For some projects the analysis computes a fast cohort at query time, and then there is no lag. People who are pinned today and not yet in the stored cohort count in the pinned arm. Exclude on the person property itself in both places.

Recommend: if they need to compare cohorts, run separate experiments per cohort, or use a single random assignment and analyze the cohorts as breakdowns of the same experiment (with the multiple- comparisons caveats from references/interpretation.md). If the override exists by accident (left over from QA / pre-launch validation), remove it: set variant: null on the affected release group, or delete the group entirely. On a young experiment with little accumulated data, reset + relaunch after the edit; on an experiment with significant clean data from before the issue was noticed, treat the post-launch window as contaminated and consider end + relaunch.

A7b — A forced-variant group starves the other arm [HIGH]

The cohort-vs-cohort case above invalidates significance but still collects both variants. A worse shape is a forced-variant release group whose properties are broad (or empty) at high rollout: it captures most or all of the population, so the other variant receives almost no analyzable exposures. Two shapes:

  • Unconditional catch-all forcing one variant. A release group with empty properties[] (matches everyone) and a pinned variant at rollout_percentage: 100. Every user who doesn't match an earlier, narrower group falls through to it and is forced to that variant; the randomized multivariate split never applies. One arm is orders of magnitude larger than the other, and the small arm holds leftovers from earlier flag versions.
  • "All new users" cohort forcing one variant, with no control path. A release group like created_at_unix >= <ts> → variant: test at 100%, where no release group leaves variant: null and no group forces the other variant. Every new account is forced to test; the control arm stops receiving new assignments and starves over time. Control collapses from a balanced share to a trickle within weeks of the forced group being added, while test keeps growing.

Detect (config-only, from experiment-get). Enumerate feature_flag.filters.groups[] and for each read variant, properties (an empty array = catch-all matching everyone), and rollout_percentage. Red flags, any of:

  • a group with variant set and broad/empty properties at high rollout_percentage;
  • no group with variant: null or with a variant that is not one of the flag's variant keys, i.e. nothing is randomized at all;
  • every variant-pinned group forces the same variant — i.e. there is no release path to the other arm.

No banner covers this shape. The single-variant banner of A5 reads the variant split, and a pinned group leaves the split untouched.

Confirm from exposures, and use the trend to read intent. exposures.total_exposures shows the starved arm immediately as one variant's persons being orders of magnitude below the other. Then read the daily series (exposures.timeseries, cumulative by the day of first exposure); the shape tells you what happened and is worth pulling before you characterize it:

  • Ran balanced, then one arm collapses — both arms roughly even for a period, then one variant's new assignments drop toward ~0 from a specific date. The experiment ran as a real A/B and was then rolled out via the flag. The balanced window is the valid result.
  • One arm never received meaningful traffic — the minority variant is ≈ internal pins / a trickle from the start, never a real share. It was served one variant from the start; it likely never ran as a randomized A/B at all.

This is distinct from the diagnostic-snapshot "plateau" (where the application stopped firing the flag) — here the app still fires; the flag config forces the variant, so the cause is visible in feature_flag.filters.groups[], not just the event stream.

Calibrate before reporting — this usually mirrors a rollout, not a bug. A broad set forcing a variant at 100% is most often a deliberate rollout done through the flag instead of the experiment UI (or a default being forced), with the experiment left in running status — not an accident. Two things sharpen the read:

  • Which variant is forced. Forcing test (the new behaviour) = the new feature was rolled out to everyone. Forcing control (the status quo) = the default was served to everyone, i.e. the feature was effectively not shipped — worth surfacing as a question, since it's easy to pin the wrong variant ("did you intend users to get the new experience, or the status quo?").
  • The exposure trend above — ran-then-rolled-out vs never-randomized.

Whatever the intent, while the flag forces a variant the experiment cannot produce a valid control-vs-test readout, and its results page should not be read as an A/B. Recommend concluding the experiment (read any pre-rollout balanced window as the result); if it was genuinely accidental, removing the forced-variant group(s) and resetting restores randomization. Surface the finding and confirm intent rather than asserting the experiment is "broken" (consistent with Step 4's don't-assume-intent guidance in SKILL.md).

A8 — Migrating the distinct_id strategy during a running experiment [HIGH]

If the user is changing how distinct_id is sent (e.g. anonymous → identified user ID, or email-as-ID → stable user ID, or a different identifier altogether) while an experiment is running, the affected people are drawn again the next time the flag is evaluated. The variant comes from a hash of the flag key and the identifier the flag buckets on. For a flag that buckets on the person, without persistence across authentication, that identifier is the distinct_id: a new id is a new draw, and on a 50/50 split about half of the affected people land in the other variant.

Not affected: a flag that buckets on the device id, and a group-aggregated flag. Their identifier does not change.

What the data shows. When the old and the new id are linked to one person, the person counts as $multiple. When they are not linked, the analysis sees a second person, and nothing flags it. The migration is a change in the user's code, so no activity log shows it.

Recommend:

  • Finish or end the running experiment before the identifier migration, then start a fresh experiment under the new strategy.
  • If they have to migrate during the run, expect inflated $multiple and treat the affected window as contaminated — use reset + relaunch once the migration is complete.
  • An "experience continuity" / flag-persistence approach can paper over anonymous → identified transitions but is not a general substitute for the migration above (see A3 tradeoffs).

A9 — Exposure events are lost or sent unevenly between the arms [MEDIUM]

Symptom (in the user's vocabulary): "the split is 58/42 and should be 50/50", "one variant gets fewer users every day", "our own analytics show an even split and PostHog does not". Often with the mismatch warning, sometimes with a p-value that is low and not yet below the threshold.

Behavior. The flag assigns evenly. The exposure event does not arrive evenly, because the arms differ in what happens around the moment the event is sent:

  • One arm navigates away. The variant redirects or reloads the page right after the flag is read, and the browser leaves before the SDK sends its queue.
  • The exposure sits behind the divergence. The event is sent on a page, in a component or after a step that one arm reaches more often, later or never.
  • One code path reads the flag early. The code reads the flag before the SDK has loaded the flags and falls back to a hardcoded variant, or it reads the flag through a call that sends no exposure (B1 in empty-experiment.md). In posthog-js an early read returns nothing only when no flags are cached from an earlier page load, so this loss falls on people with no cached flags, mostly first visits.
  • One arm is slower or fails more often, so people leave before the event is sent.
  • A cached page carries the assignment [LOW]. A page that is cached with a variant or an identifier inside hands one assignment to many browsers.
  • The exposure is recorded before anything is rendered. A server or an edge function records the exposure, and a share of those requests never renders a page: prefetches, bots, blocked browsers. The split of exposures is even, and the arms still differ in who arrives. The sample ratio test does not see this shape.

Evidence to gather.

  • Where the gap sits. Exposures per variant by day and by library (query in diagnostic-snapshot.md). A gap in one library, or from one day on, names the code path or the deploy. The same query by page shows an exposure that one arm sends from a page the other arm never reaches.
  • The exposure criteria first. A filter in exposure_criteria.exposure_config.properties that only one arm's events can match gives the same picture, and is a setting, not code (B12 in empty-experiment.md).
  • New against returning people. A gap among returning people only has two candidates: this entry, and the SDK that does not send the event again for a person who already sent it (B3 in empty-experiment.md). No tool splits exposures this way: add person.created_at < toDateTime('<start_date>', 'UTC') AS returning to the SELECT and the GROUP BY of the "By day and library" query in diagnostic-snapshot.md, and compare the arms within each group. B3 removes returning people from every arm in proportion, so a gap in one arm points here.
  • With a custom exposure event: the split of the flag calls against the split of the custom event. Even flag calls and an uneven custom event put the loss between the flag read and the exposure.
  • The code around the flag read in each arm: a redirect, a reload, a fallback value, a different component. This is a question for the user when the repository is not at hand.

Co-occurs with: A2 (this is its most frequent cause), A10, and B8 in empty-experiment.md.

Recommendation. Send the exposure at a point that both arms pass before they diverge.

  • A custom exposure event at the common entry, for example the page view of the landing URL. It needs the variant in $feature/<flag-key> (B4 in empty-experiment.md).
  • An exposure from the server at assignment, when the arms diverge in the browser.
  • Wait for flags to load before the first read, and send no fallback variant.

The results collected so far carry the bias. After the fix, reset and relaunch, or move the start of the analysis to the fix.

A10 — Part of the audience gets no variant from the flag [MEDIUM]

Symptom (in the user's vocabulary): "most responses on the flag are false", "users saw the new UI for a second and then lost it", "far fewer exposures than visitors on that page", "control is much bigger than it should be".

Behavior. Only an exposure whose response is one of the flag's variant keys counts. A response of false, an empty response, or a key that is no variant is no exposure, and nothing warns about it. The people who get no variant are rarely a random share, so the people who stay are a biased sample, and the arms can lose different shares.

Why an evaluation returns no variant:

  • The flag's inputs are not there yet. A release condition reads a person property that the backend sets (a sign-up date, a plan, an app version). A client that evaluates before the property is stored cannot match. New users in onboarding are hit most. With person_profiles: 'identified_only', anonymous visitors have no stored person properties: the flag sees only what the client sends with the call.
  • Client and server hold different values (A4, second shape).
  • Device bucketing without a device id. A flag that buckets on the device id returns false for a call that carries none, which is the usual case for a server-side call. A release condition at a rollout of 100% that pins a variant still matches without a device id, so such calls carry only the pinned variant.
  • A group-aggregated flag without the group. A flag that is aggregated by a group type returns false for a call that carries no key for that group type.
  • The flag is read before flags load, or before a consent step lets the SDK load them. In posthog-js the read returns nothing only when no flags are cached from an earlier page load: a retu