AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

diagnosing-experiment-health

Runs diagnostics and a health check on one PostHog experiment: its setup, exposures, results and changes during the run. Covers 0 exposures, sample ratio mismatch, lost or uneven exposures, users in multiple variants, identity faults, who the experiment counts, significance traps (peeking, A/A, 'target reached', Bayesian vs Frequentist, sequential, CUPED), mid-run edits, pause and freeze, and survey follow-up. TRIGGER when: user asks 'is my experiment healthy / set up right / biased?' or 'why 0 exposures?', asks why it counts different people than their insight, mentions the bias or mismatch warning, an uneven split, significance that flips, numbers that disagree with their SQL, results that changed after an edit, or wants user feedback on an experiment. DO NOT TRIGGER when: creating an experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle).

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--diagnosing-experiment-health at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 8

references/interpretation.md · 24000

is set. A 30% effect on a conversion rate needs a small sample, so the target is reached within days on a busy surface. A summed value such as revenue varies much more from person to person, so its target can be many times larger.

  • In manual mode the target comes from the baseline the user typed, not from the results.
  • "Target reached" says that an effect of that size would probably have shown. It says nothing about significance, and it does not end the experiment.
  • The running time settings list the exposures and the change that the experiment can detect today, at the next week marks, and on the date it reaches the target. A change over 100% shows as ">100%".

Evidence to gather. running_time_calculation in experiment-get:

  • recommended_sample_size — the target, a total across all variants.
  • recommended_running_time — the days still missing at the current exposure rate, and 0 once the target is reached. A value far beyond the time the team will run the experiment (many months, years) means that the target is out of reach at this traffic: the experiment cannot detect an effect of that size in a useful time (C8).
  • exposure_estimate_config.conversionRateInputType — manual means that the user typed the baseline and the daily exposures. Both numbers above then come from that input: recommended_running_time is the whole estimated length, not the days left, and neither number is updated during the run. The same holds for numbers that were written when the experiment was created, through the API or by an agent, until someone opens the launched experiment. A recommended_running_time above 0 on an experiment whose exposures have passed the target is one of these cases, or a value saved by a page load before the target was reached.
  • minimum_detectable_effect — the experiment's own value, or the project default that was copied in when the experiment was created. When it is absent the project default applies. experiment-setup-context returns that default (team_defaults) where the tool is available. Otherwise ask the user.

In automatic mode the product estimates the target again from the experiment's own data whenever someone opens the page of a launched experiment, and saves it. So the target moves during the run, and the label a user saw last week can differ from today's stored numbers. Compare the minimum detectable effect with the effect the user hopes for.

Co-occurs with: C1 (the same people read results early), C2.

Recommendation.

  • Explain the label first: a target for sample size, not a verdict.
  • When the minimum detectable effect is far above a realistic effect, the target is too small. Lower it to the smallest effect worth shipping for: the required sample grows with the inverse square, so half the effect needs four times the sample.
  • When the decision metric is not the first primary metric, the estimate is for the wrong metric.
  • The rule of thumb behind the calculator: the sample per variant is about 16 × variance / (absolute effect)². For a conversion rate the variance is p × (1 − p).

C13 — Statistics settings that change how a result reads (sequential testing, CUPED, baseline variant) [HIGH]

Three settings change what the numbers on the page mean. Each is set on the experiment. Sequential testing and CUPED fall back to the project default where the experiment does not set them. The baseline falls back to control.

Sequential testing (stats_config.frequentist.sequential_testing_enabled). Frequentist only.

  • The p-value and the interval stay valid however often the result is read. C1's peeking warning does not apply.
  • The price is a wider interval than a fixed-horizon test gives on the same data. "The calculator says significant and PostHog does not" is expected (C11).
  • The interval narrows as the sample grows. Its extra width over a fixed-horizon interval is smallest when the two arms of a comparison together hold about the tuning parameter's sample size (sequential_tuning_parameter, default 5,000 or the project's default), and larger far from it. An experiment with a much larger sample and the default value has a wider interval than it needs.
  • A p-value of exactly 1.000 is normal [MEDIUM]: the value stays at 1 until the evidence crosses a threshold.

CUPED (stats_config.cuped.enabled).

  • It adjusts the metric values with each person's behavior before exposure, and narrows the interval when that behavior predicts the metric.
  • It does not change who is in the experiment or which events count.
  • The effect and the interval come from adjusted values, so no query on raw events and no calculator reproduces them (D1 in numbers-vs-sql.md).
  • It applies to mean metrics and to ordered funnels. Ratio and retention metrics are never adjusted. A mean metric with a threshold, a mean metric on a session property (such as $session_duration) and a funnel with a data-warehouse step are not adjusted either.
  • It reads a look-back window before each exposure (14 days by default), which makes the calculation heavier.

Baseline variant (stats_config.baseline_variant_key).

  • Every comparison is against the baseline. That is control, or the flag's first variant when it has no control, unless the experiment names another variant.
  • After a change of the baseline every delta, interval and significance is recalculated against the new one, and the baseline itself shows no delta. "Control shows a lift" and "all signs flipped" are this setting.

Evidence to gather. stats_config in experiment-get, and the experiment's change history when a result moved without new data: a setting changed on the page recalculates the whole run (D8 in numbers-vs-sql.md). What a missing key means depends on the key:

  • A missing method reads as Bayesian, and a missing confidence level as 95%. The project's defaults for both are copied into an experiment when it is created.
  • frequentist.sequential_testing_enabled and cuped fall back to the project's current default.

experiment-setup-context returns the project's defaults (team_defaults) where that tool is available. Otherwise ask the user to read the project's experiment settings. A p-value of exactly 1.000 on a metric of a Frequentist experiment is a sign of sequential testing. A p-value below 1 does not rule it out.