AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

debugging-experiments

Debug and support PostHog Experiments (A/B tests) for a customer looking at their own results. Use whenever an experiment support ticket is pasted or a customer asks a results question, most commonly "why aren't my exposures even?", "why is one variant getting no traffic?", "why am I missing / seeing too few exposures?", "why does the bias banner show?", or "why don't PostHog's numbers match my SQL?". Pulls the experiment's real data read-only, matches it to a known-cause catalog, and produces a customer-facing explanation, fix, and review of the pertinent numbers. Loads diagnosing-experiment-health as its deep diagnostic library. DO NOT TRIGGER when: creating an experiment (use creating-experiments), only configuring rollout (configuring-experiment-rollout) or metrics (configuring-experiment-analytics), asking lifecycle questions (managing-experiment-lifecycle), or the underlying feature flag is what's misbehaving rather than the results (use debugging-feature-flags).

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read posthog--debugging-experiments at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 4

references/real-vs-noise.md

Is a downstream effect real or noise?

Symptom: a funnel step the feature doesn't touch (e.g. event page → checkout) shows a lift, often while the step it does touch is flat. The customer suspects a measurement bug.

Why that leg isn't a clean comparison

A rate measured between two mid-funnel steps only counts users who already reached the earlier step, and reaching it happens after randomization and can be nudged by the treatment. So the leg compares two groups shaped by the experiment, not the randomized groups, and a gap can appear there with no real effect on the downstream action. It can even read more significant than the honest metric, because it's a smaller, more-selected denominator.

Trust the randomized endpoint — exposure → final step, counting everyone assigned — over any rate measured between two mid-funnel steps. PostHog computes significance from the first step to the last step for exactly this reason (see numbers-vs-sql.md D2 in the diagnosing-experiment-health library).

Three checks — any one failing points to noise

  1. Non-user split. Recompute the effect among users who never fired the feature-interaction event. If the between-variant gap is still there, the feature can't be causing it. State it as a between-variant comparison restricted to non-users — not "non-users convert more".
  2. Dose-response. Rank the variants by actual feature usage. A real effect is strongest where the feature is used most; an effect that's absent in the high-usage arm and present in a barely-used arm is not causal.
  3. Cohort stability. Split the outcome by week of first exposure. A real effect holds its sign and rough size across cohorts; one that flips ahead/behind or lives in a single cohort is noise.

Recommend

Report the randomized exposure → outcome number (with its win probability) as the verdict, explain the conditioning trap in plain terms, and — if the checks point to noise — advise against shipping on the downstream figure and to keep running to the pre-planned sample.