AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

diagnosing-experiment-health

Runs diagnostics and a health check on one PostHog experiment: its setup, exposures, results and changes during the run. Covers 0 exposures, sample ratio mismatch, lost or uneven exposures, users in multiple variants, identity faults, who the experiment counts, significance traps (peeking, A/A, 'target reached', Bayesian vs Frequentist, sequential, CUPED), mid-run edits, pause and freeze, and survey follow-up. TRIGGER when: user asks 'is my experiment healthy / set up right / biased?' or 'why 0 exposures?', asks why it counts different people than their insight, mentions the bias or mismatch warning, an uneven split, significance that flips, numbers that disagree with their SQL, results that changed after an edit, or wants user feedback on an experiment. DO NOT TRIGGER when: creating an experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle).

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--diagnosing-experiment-health at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 8

references/bias-and-skew.md · 48000

rning visitor gets the cached value.

  • The application's own fallback [MEDIUM]. Code that reports "no variant" as control puts everyone outside the rollout into control, in every count that the application fills: the user's own analytics, and a custom exposure event that carries the application's value. Control there holds every person the flag gives no variant, on top of its own share of the people with a variant. The default exposure event is not affected: its response stays false and is no exposure.
  • A holdout. People in a holdout get holdout-<id> in place of a variant. That is intended: they are outside the experiment.

Evidence to gather.

  • The responses of $feature_flag_called by day and by library ("By day and library" in diagnostic-snapshot.md, with $feature_flag_called as the event): the share that is no variant key, and whether it sits in one library. $experiment_exposure holds no false or empty response, so it cannot show these rows.
  • feature_flag.filters.groups[].properties: a condition on a property that is set after sign-up, or by the backend.
  • feature_flag.bucketing_identifier.
  • A false share that equals what the release conditions exclude by design is not a finding.

Co-occurs with: A2, A4, A9, and B10 in empty-experiment.md.

Recommendation, in order of preference:

  1. Send the exposure from the place that knows the variant the user got.
  2. Put the variant on the exposure event explicitly.
  3. Make the flag's inputs available before the evaluation: pass the person properties with the call, and wait for flags to load.
  4. Avoid release conditions on data that does not exist at evaluation time.