AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

diagnosing-experiment-health

Runs diagnostics and a health check on one PostHog experiment: its setup, exposures, results and changes during the run. Covers 0 exposures, sample ratio mismatch, lost or uneven exposures, users in multiple variants, identity faults, who the experiment counts, significance traps (peeking, A/A, 'target reached', Bayesian vs Frequentist, sequential, CUPED), mid-run edits, pause and freeze, and survey follow-up. TRIGGER when: user asks 'is my experiment healthy / set up right / biased?' or 'why 0 exposures?', asks why it counts different people than their insight, mentions the bias or mismatch warning, an uneven split, significance that flips, numbers that disagree with their SQL, results that changed after an edit, or wants user feedback on an experiment. DO NOT TRIGGER when: creating an experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle).

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--diagnosing-experiment-health at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 8

references/mid-run-changes.md · 24000

experiment-duplicate rejects a legacy experiment. Alternatively, end the existing experiment with a documented conclusion if the original hypothesis is no longer interesting.

E14 — Cleaning up the flag after the experiment [HIGH]

What the link between an experiment and its flag allows:

  • Deleting the flag is blocked only while the experiment runs. A draft, stopped or archived experiment does not block it, and the experiment keeps its results.
  • A deleted flag frees its key. Restoring it later reclaims the key, or gets a numeric suffix (my-flag-2) when another flag took the key meanwhile. The application code then reads a key that is no longer this flag.
  • The flag cannot be unlinked from the experiment.
  • Simplifying the flag [MEDIUM]. Once the experiment has ended, the flag no longer protects its variant keys, so the variants can be removed on the flag itself. The ended experiment reads its variant keys from the flag: a removed variant leaves its results at the next calculation. On the flag's overview the edit control of the variants stays disabled while an experiment is linked. The flag's edit form is the way in.
  • Archiving needs an ended experiment.

Results that seem to be gone. Deleting an experiment hides its results. Deleting its flag does not: the experiment keeps its results. Neither deletes an event. A deleted flag can be restored from its page. A deleted experiment can be restored too: the delete confirmation offers an undo, and an API update that sets deleted to false restores it, once its flag is restored. The app has no list of deleted experiments, and experiment-update does not take deleted.

Recommend: before cleaning up, confirm the flag's future use. If the user expects to keep using the flag for general rollout after the experiment ends, ship the variant (E7) — that leaves the flag in a usable state at the chosen rollout. If they're done with the flag too, keep the flag until the calling code has been removed.

E15 — Restarting an experiment with new variants [MEDIUM]

The clean approach is:

  1. End the existing experiment.
  2. Duplicate it (experiment-duplicate), or create a new one. The duplicate is a new draft with the metrics and settings of the original, without its holdout and its excluded variants.
  3. Give it a new feature flag key. A duplicate without a new key shares the original's flag, so a ship or a pause on one affects both.
  4. Launch the new experiment under the new flag.

Why a new key and not a reset on the old flag:

  • The same key gives every returning person the same draw as before, so the new run is not a fresh randomization for them.
  • Returning people who already sent the exposure event may not send it again (B3 in empty-experiment.md).
  • A changed set of variants on the old flag moves people between variants (E3).

The same three points explain a follow-up run that disagrees with the first one on the same flag: it is not an independent test.

Reusing the same flag with new variants on a new experiment is technically possible but tends to produce confusing exposure histories. Only do this if the user is explicit about wanting to keep historical bucketing comparable.

E16 — Freezing and unfreezing exposure [HIGH]

Symptom (in the user's vocabulary): "exposures stopped growing and the experiment still says running", "after the freeze some users lost the variant", "can I freeze this experiment", "the freeze was refused".

Behavior. Freezing stops enrollment and keeps measuring. PostHog saves the people exposed so far as a static cohort and narrows every release condition of the flag to that cohort. The variant split and the end date stay untouched. status reads exposure_frozen, and no banner appears.

  • New people get no variant.
  • Enrolled people keep their variant, and their metric events keep counting.
  • The cumulative exposure series goes flat. That is the intended effect.

Where a freeze biases or breaks the run:

  • People exposed just before the freeze can miss the cohort. They count as exposed, and from the freeze on they get no variant.
  • Local evaluation. A static cohort cannot be evaluated locally. A server SDK falls back to a remote call, and one that is set to evaluate only locally serves the flag's default to everyone, enrolled people included [MEDIUM].
  • Release conditions edited or added after the freeze. A variant override set later on a frozen release condition keeps that condition's cohort, so people outside the cohort still get no variant from it. An override set on a new release condition carries no cohort: that is the next case. A release condition added without the cohort reopens enrollment, and status reads running again.
  • Ending without shipping leaves the flag narrowed to the cohort. Shipping a variant or resetting removes the freeze.
  • Unfreezing reopens enrollment. People who enroll after it joined later and possibly under other conditions than the first group. Mixing both groups in one analysis can bias the result. The unfreeze dialog is the one place where the product itself warns about bias. The End dialog of a frozen experiment warns only that the flag stays narrowed to the cohort.

When a freeze is refused: a draft, an ended or a paused experiment; an experiment that is already frozen; a missing or deleted flag, or a flag without release conditions; a group-aggregated experiment; an experiment with a holdout or with early-access conditions in the older format (super_groups in the flag's filters); too many exposed people to save in one step; more than a small share of exposed people without a person profile, which rules out experiments on logged-out traffic; no exposures yet.

Evidence to gather. status from experiment-get. Every freeze and unfreeze in experiment-activity, with its time: a change of the flag's release conditions that adds the cohort to every condition, or removes it. Newer ones also have a named entry on the experiment (exposure_frozen, exposure_unfrozen). Older ones have only the flag's change, which is missing when the caller has no access to the flag. Any other change of the flag's release conditions after the freeze. Whether the user's servers evaluate flags locally.

Co-occurs with: E9, E12, and the plateau check in empty-experiment.md.

Recommendation.

  • A plateau on a frozen experiment needs no fix.
  • State the freeze and unfreeze dates with any result that spans them.
  • Before an unfreeze, say that the two enrollment groups will be mixed. A user who only wanted to check the frozen results does not need to unfreeze.
  • For a flag with many release conditions or overrides, freezing is fragile: every later edit of the flag has to keep the cohort condition.

E17 — The start date was changed after launch [MEDIUM]

Symptom (in the user's vocabulary): "the numbers changed overnight and nobody touched the experiment", "people from launch day disappeared", "the experiment and the insight disagree since last week".

Behavior. The experiment analyzes exposures from start_date on. An edit of the start date moves that boundary for the whole run:

  • Moved later: people whose exposures all lie before the new start leave the experiment. A person exposed before and after counts from the first exposure inside the window, and their earlier metric events no longer count.
  • Moved earlier: exposures from before the launch enter: internal testing, another split, another audience.

An insight on the same events ignores the experiment's start date, so the two disagree after such an edit.

Evidence to gather. Changes of start_date in experiment-activity, each with its old and new value. A start date before the flag had its current split or audience (the flag's entries in the same history).

Co-occurs with: E3, E8.

Recommendation. A start date moved to just after a fix is a legitimate way to cut a contaminated beginning: say what was cut. The user makes this edit on the experiment page; no tool changes the start date. A start date moved back to recover data brings in whatever the flag did then: check the flag's history for that period before trusting the result. Repeated edits of the start date are a reason to reset and relaunch.