AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

diagnosing-experiment-health

Runs diagnostics and a health check on one PostHog experiment: its setup, exposures, results and changes during the run. Covers 0 exposures, sample ratio mismatch, lost or uneven exposures, users in multiple variants, identity faults, who the experiment counts, significance traps (peeking, A/A, 'target reached', Bayesian vs Frequentist, sequential, CUPED), mid-run edits, pause and freeze, and survey follow-up. TRIGGER when: user asks 'is my experiment healthy / set up right / biased?' or 'why 0 exposures?', asks why it counts different people than their insight, mentions the bias or mismatch warning, an uneven split, significance that flips, numbers that disagree with their SQL, results that changed after an edit, or wants user feedback on an experiment. DO NOT TRIGGER when: creating an experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle).

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--diagnosing-experiment-health at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 8

references/bias-and-skew.md

Bias & skew on a running experiment

Variant counts don't match the configured split, one variant looks biased, the in-app warning banner appeared, or users are showing up under multiple variants.

Before diagnosing

Read three values first. The first two come from the exposure data of the Step 1.5 snapshot (exposures in experiment-results-get), the third from experiment-get. experiment-results-get runs queries: call it once, after the stored results and the change history of the snapshot, and reuse its exposures for every check in this file. The first two are computed on the population the experiment analyzes, which a query of your own does not reproduce.

  1. The sample ratio test: exposures.sample_ratio_mismatch.p_value. The page flags a mismatch below 0.001.
    • Below 0.001: a real assignment or capture problem, or a split that changed during the run, because the test compares the whole run with the current split. Walk A2.
    • At or above 0.001: no mismatch by the product's rule. On a small sample the visible split is noise (see C2 in interpretation.md).
    • A value between 0.001 and 0.05 is not cleared by that rule [MEDIUM]. A slow leak of exposures in one arm looks like this for days before it crosses the threshold. No tool returns the p-value of an earlier day. Read the series in exposures.timeseries, which is cumulative per variant: take the difference between days, and walk A9 when the gap grows in one direction.
  2. People in more than one variant: exposures.total_exposures['$multiple'] over the sum of all values. Non-zero puts A1, A3 and A4 on the table, and A6 and A8 when the flag or the identifier changed during the run. Under first_seen handling the key is absent, because each person keeps the first variant.
  3. The configured split. feature_flag.filters.multivariate.variants[].rollout_percentage from experiment-get. An uneven split amplifies whichever bias source is present. The flag shows the split of today. On an experiment that has ended, or whose flag was edited, take the split of the run from the change history ("Changes after launch" in diagnostic-snapshot.md).

When the observed split matches the split the flag had during the run, there is no mismatch to explain. A user who expects 50/50 on a flag that ran at 90/10 has a question about the configuration, not about bias: say which split the flag had and since when. If the split changed after launch, walk E3 in mid-run-changes.md. The product's sample ratio test compares against the flag's current split, so after such a change its p-value tests the wrong expectation.

If the symptom is "metric count is far smaller than exposures" (e.g. 10× or 100× gap), walk this file before numbers-vs-sql.md — that shape of divergence is most often a bucketing / identity problem (A3/A4), not a query-scope problem.

The SQL in this file follows the query rules in diagnostic-snapshot.md: the experiment's exposure event, the flag's variant keys, a window with both ends set, and no test-account filter. The exposure event is resolved_exposure_event, unless exposure_criteria.exposure_config names $experiment_exposure: then it is that event ("Which event, which property" in diagnostic-snapshot.md). When resolved_exposure_event is $feature_flag_called, also read $experiment_exposure (event IN ('$feature_flag_called', '$experiment_exposure')), and count persons, not rows. The queries count person_id. On a flag aggregated by a group type (feature_flag.filters.aggregation_group_type_index is set), put $group_<index> in its place and leave out the rows where it is empty. When a result is compared with $multiple, leave out the keys in excluded_variants: the experiment never counts them. resolved_exposure_event is always the default event. On an experiment with a custom exposure event these queries read the flag's calls, not the people the experiment counts: say so when you report them.

Contents

  • A1 — Multi-variant exclusion bias on uneven split
  • A2 — Sample ratio mismatch (SRM)
  • A3 — Identity fragmentation (users in both control and test)
  • A4 — Bootstrap × /flags variant disagreement
  • A5 — Flag/experiment state inconsistency
  • A6 — Mid-run flag edits that rebucket already-exposed users
  • A7 — Non-randomized assignment via release conditions
  • A7b — A forced-variant group starves the other arm
  • A8 — Migrating the distinct_id strategy during a running experiment
  • A9 — Exposure events are lost or sent unevenly between the arms
  • A10 — Part of the audience gets no variant from the flag

A1 — Multi-variant exclusion bias on uneven split [HIGH]

This is the signal behind the in-app warning "The uneven split may bias the results".

Evidence. The banner's check reports a finding when all four hold:

  • multiple_variant_handling == "exclude" (the default)
  • the variant rollouts are uneven. A variant at 0% counts, so a flag at 0/50/50 is uneven.
  • the share of people in more than one variant is above 0.1%. The value is exposures.bias_risk.multiple_variant_percentage: the $multiple count over all exposures.
  • the experiment has no end date. After the end the check reports nothing, and the banner is gone. The exclusion is still in the numbers: the banner's absence on an ended experiment clears nothing.

The warning-vs-visible gap. The banner fires above 0.1%. The collapsed summary of the Exposures panel leaves the $multiple row out at or below 0.5%. The expanded table always lists it, labeled "(excluded from analysis)". So a user can see the banner while the summary shows a clean split. The warning is real: point the user to the expanded table, and state the exact share from exposures.bias_risk.

Mechanism. Multi-variant users are dropped, but the smaller variant loses a larger fraction of its assignments than the larger variant. Multi-device / multi-session / signup-flow users tend to be high-intent — so the smaller variant keeps a low-intent slice and looks worse than it should. This is asymmetric exclusion bias, not a UI bug.

Recommend (in this order):

  1. Switch to an equal split. See configuring-experiment-rollout. On a draft experiment this is free. Mid-run it's an anti-pattern — prefer reset or end+restart over changing the split mid-run. The banner's own "Adjust distribution" button opens the split editor on the running experiment. A user who followed it changed the split mid-run: read the flag's history before any other cause (E3 in mid-run-changes.md).
  2. Switch multiple_variant_handling to "first_seen". See configuring-experiment-analytics. Mid-run this is the low-disruption option — no users switch variants, all already-collected data stays in the analysis. first_seen is less biased than exclude for this specific shape, not unbiased: it counts the first variant a user saw and ignores later ones, which still asymmetrically discounts engaged multi-session users. There is no clean fix for the underlying problem; the trade-off is between which bias the user prefers.

The experiment's settings and the public docs call Exclude the recommended handling. That holds for an even split, and a user may quote it: the banner is the product's own exception for uneven splits.

A2 — Sample ratio mismatch (SRM) [HIGH]

Evidence. exposures.sample_ratio_mismatch holds the p-value of a chi-squared test and the expected count per variant.

  • The test runs once the variants with a rollout above 0% hold 100 exposures together.
  • With fewer than two variants above 0% there is no test, and the value is null. That is the state after a ship and after a split edited to 100/0. Judge such an experiment by the split of the run ("Changes after launch" in diagnostic-snapshot.md).
  • The page flags a mismatch at p < 0.001. The server returns the p-value and applies no threshold.
  • The expected counts follow the flag's split as it is now. They are rescaled for a holdout. $multiple, the holdout and excluded variants are left out.
  • After a split change the test compares the whole run with the new split, so a mismatch is the expected reading. Read A6 before anything else in that case.

MUST read the product's value. MUST NOT run the test on event counts of your own: raw exposure events include test accounts, responses that are no variant, repeated events of one person and events outside the analysis window.

What it means. The distribution of exposed people differs from the flag's current split by more than chance allows. Either the split changed during the run (cause 1), or something biases the assignment or the capture of exposures. A visible gap on a small sample is not a mismatch until the test says so, and a mismatch on a small sample is still a mismatch: the test already accounts for the sample size. The exception is a single reading that clears on the next day's data, which happens about once in a thousand readings by construction.

Causes, cheapest evidence first. Surface each cause the evidence supports (Step 3 of SKILL.md).

  1. The flag changed after launch. Evidence: a change of the flag's filters after start_date in experiment-activity. Which edits move people, and which do not: A6.

  2. Exposure events are lost or sent unevenly between the arms [MEDIUM]. The flag assigns evenly and the event does not arrive evenly: a redirect, an exposure placed after the arms diverge, a flag read before flags load. It is the most frequent cause of a mismatch that no flag change explains. See A9.

  3. Part of the audience gets no variant. Responses of false or an empty value are no exposures, and the people who drop out are not a random share. The flag decides whether a person gets a variant apart from which variant, so a false from the flag removes people from every arm alike. It moves the split only when something ties the loss to one arm, such as the application's own fallback on a custom exposure event, or a pinned condition that still matches without a device id (A10). See A10.

  4. A release condition pins a variant. Evidence: a variant in feature_flag.filters.groups[] that is one of the flag's variant keys. See A7.

  5. Server and client evaluate the flag differently [MEDIUM]. A bootstrapped value against a later evaluation (A4). A server that evaluates locally keeps the old definition for its polling interval after a flag edit. A flag with persistence across authentication is not evaluated locally: the SDK falls back to a remote call, so the two paths can answer at different times with different inputs. With strictLocalEvaluation or onlyEvaluateLocally set in posthog-node there is no remote call, and the call returns no value, which is no exposure. Evidence: exposures per variant by library (snapshot query), feature_flag.ensure_experience_continuity, and whether the user's servers evaluate locally.

  6. Bots on server-side evaluation [LOW]. The public docs rank this first. As a cause of a directional mismatch it is weak: one crawler is one identifier and counts once. It matters when a crawler sends a new identifier with every request. Evidence: the share of server-side libraries among the exposures, and exposed people with no later event. Fix path: the "Filter Bot Events" transformation (a data pipeline transformation) applies its built-in bot lists to browser events only. For events from a server SDK it filters only what the user adds as custom user-agent patterns or IP prefixes. With "Keep events where the useragent is not set?" at No, it also drops every event without the user-agent property ($raw_user_agent by default), and posthog-node sets that property only when the code passes it. The more reliable fix is to not evaluate the flag for requests the server already knows to be bots.

What does not cause a mismatch.

  • Repeated exposure events of one person. The experiment counts each person once, at the first exposure. A server SDK that sends the event again after a restart or after its dedup cache fills up changes the event count, not the split.
  • Ad-blockers and network loss. Both arms lose alike. The sample shrinks, and a reverse proxy recovers part of it.
  • A holdout. The expected counts are rescaled for it.
  • People in more than one variant, on an even split. They leave both arms alike. On an uneven split see A1.

Fix paths change the flag. An agent can edit a flag over MCP. On a running experiment every such edit is a mid-run change (A6, and Step 4 of SKILL.md): explain the effect and get the user's decision before any edit.

A3 — Identity fragmentation (users in both control and test) [MEDIUM]

This is an identity problem, not a bias problem. One human has two or more distinct ids. Each id is hashed by itself, so each can land in a different variant.

  • When the ids are linked to one person, the analysis sees one person with two variants and counts the person as $multiple.
  • When the ids were never linked, the analysis sees two persons with one variant each. Nothing in the exposure data shows it.

Evidence.

  • The count. exposures.total_exposures['$multiple']. No event carries $multiple. The analysis computes it per person, and only from the flag's current variant keys: a response of false, an empty response or an old key never makes a person $multiple.

  • Under first_seen handling the count is not reported. The query below is then the only source.

  • Ids per person. In the raw-responses query of the snapshot, distinct_ids / persons above 1 for a variant means that people evaluated the flag under several linked ids. That is the precondition for a flip. It says nothing about ids that were never linked.

  • The people, when the cause has to be named. Start with one day on a large project. The query finds only people with two variants inside its window, so an empty result on a short window does not rule out A3:

    -- People whose exposure events carry more than one variant key
    SELECT
        person_id,
        uniq(properties.$feature_flag_response) AS variants_seen,
        uniq(distinct_id) AS distinct_ids,
        groupUniqArray(properties.$feature_flag_response) AS variants
    FROM events
    WHERE event = '<resolved_exposure_event>'
      AND properties.$feature_flag = '<flag-key>'
      AND properties.$feature_flag_response IN ('<variant-key-1>', '<variant-key-2>')
      AND timestamp >= toDateTime('<window_start>', 'UTC')
      AND timestamp < toDateTime('<window_end>', 'UTC')
    GROUP BY person_id
    HAVING variants_seen > 1
    ORDER BY variants_seen DESC, distinct_ids DESC
    LIMIT 20
    
  • One person's exposure events, to tell the causes apart:

    SELECT
        timestamp,
        distinct_id,
        properties.$feature_flag_response AS response,
        properties.$lib AS lib
    FROM events
    WHERE event = '<resolved_exposure_event>'
      AND properties.$feature_flag = '<flag-key>'
      AND person_id = '<person_id>'
      AND timestamp >= toDateTime('<window_start>', 'UTC')
      AND timestamp < toDateTime('<window_end>', 'UTC')
    ORDER BY timestamp
    LIMIT 50
    
    • Different distinct_ids: an identity cause from the list below.
    • The same distinct_id, and a flag edit between the two timestamps: A6.
    • The same distinct_id, no edit, different libraries: server and client disagree (A4, A10).

    A response of false has no $experiment_exposure row: when resolved_exposure_event is $experiment_exposure, run the query on $feature_flag_called to see a step to no variant.

Common causes:

  • reset() was called between sessions (other than on logout)
  • identify() ran after the flag was already evaluated
  • Cross-device usage without identity stitching
  • Cookies cleared between visits, incognito / stealth browsing
  • Anonymous → identified transition without flag persistence enabled
  • The same user has different anonymous IDs client-side vs server-side, so the flag hash bucket differs
  • Native mobile auth flows where the flag is read before the SDK identifies the user, or where authentication crosses an SDK boundary (e.g. web → in-app webview)

A note on what's fundamentally fixable vs not. Stitched-identity issues from identify() ordering, cross-domain cookies, and bootstrap timing are real bugs that can be fixed. Multi-device usage and incognito / stealth browsing are not fixable from PostHog's side — and the users who exhibit them tend to be more engaged on average, so excluding the $multiple bucket pulls a non-random slice out of the analysis. There is no clean fix; the recommendation is to contain the problem (scope exposure to the relevant flow so the denominator stays meaningful) rather than eliminate it.

Recommend:

  • Audit identify() and reset() ordering — reset() only on explicit logout, identify() before flag evaluation. Persistence across authentication carries a variant from anonymous to identified, not back: after reset() the next evaluation is a new draw.
  • For experiments spanning logged-out → logged-in flows, consider one of:
    • Persist flag across authentication steps (tradeoffs: requires person_profiles: 'always', incompatible with local evaluation and bootstrapping, adds slight latency)
    • Device-ID bucketing — appropriate for landing/marketing/anonymous flows. Keeps the variant stable across the anonymous→identified transition without the flag-persistence tradeoffs. Many users don't realize this option exists; surface it explicitly when the symptom is cross-auth bucketing. Two limits. An evaluation without a device id gets no variant (the flag returns false), which is the usual case for a server-side call that passes none (A10). The exception is a release condition at a rollout of 100% that pins a variant: it still matches, and returns the pinned variant. And it cannot be combined with persistence across authentication: the API refuses the pair.
  • For pre-auth experiments, ensure cookies/localStorage persistence is configured (cookies preferred for cross-subdomain).
  • For mobile flows, consider evaluating the flag server-side (local evaluation) once the user is authenticated rather than on first app open.

A4 — Bootstrap × /flags variant disagreement [MEDIUM]

Specific scenario: server-rendered app with bootstrapping enabled. The $multiple share in this shape can become substantial — well above the trickle you'd expect from normal cross-device traffic alone. Website-only flags (no bootstrap) are unaffected.

Mechanism. Two shapes.

  • Different identifiers. The server bootstraps flags using the server-known distinct_id, but the bootstrap payload doesn't include distinctID — so posthog-js initializes with whatever's in persistence (often the anonymous ID). The bootstrap value gets reported under the anonymous ID; then posthog-js asks the flag service with whatever ID it has after identify(). When the IDs differ, the hashes differ, and one person sends exposure events for two variants. On a flag that buckets on the device id the identifier does not change at identify(), so this shape needs different device ids.
  • Different inputs [MEDIUM]. The bootstrap values come from the user's own backend, which evaluates the flag with the properties it passes. A property in another type or format than the stored one (a timestamp as a string against a stored number), or a variant set by hand, matches on one side only. The bootstrap-sourced exposure carries a variant, and the next evaluation returns another variant or false. The user sees the new experience for a moment and then loses it.

Evidence. The exposure event carries two source properties:

  • $used_bootstrap_value — true when the value was read before the flags request of that page load answered. That is a bootstrap value, and also a value cached from an earlier page load. $feature_flag_bootstrapped_response holds what the bootstrap payload said for the flag, and is null without one.
  • locally_evaluated — true when the event came from server-side local evaluation.

The signature is one person with an event read before the flags request answered and another event for the same flag, with different variants. $used_bootstrap_value is usually not a materialized column, so this query reads the full properties of every row: keep the window to one day. The comparison with true needs the property typed Boolean in the project, which is the usual case. If the query fails on a type error, write the two conditions as toString(properties.$used_bootstrap_value) = 'true' and ifNull(toString(properties.$used_bootstrap_value), '') != 'true'.

SELECT
    person_id,
    groupUniqArrayIf(properties.$feature_flag_response, properties.$used_bootstrap_value = true) AS bootstrap_variants,
    groupUniqArrayIf(properties.$feature_flag_response, ifNull(properties.$used_bootstrap_value, false) = false) AS other_variants
FROM events
WHERE event = '<resolved_exposure_event>'
  AND properties.$feature_flag = '<flag-key>'
  AND properties.$feature_flag_response IN ('<variant-key-1>', '<variant-key-2>')
  AND timestamp >= toDateTime('<window_start>', 'UTC')
  AND timestamp < toDateTime('<window_end>', 'UTC')
GROUP BY person_id
HAVING length(bootstrap_variants) > 0
   AND length(other_variants) > 0
   AND arraySort(bootstrap_variants) != arraySort(other_variants)
LIMIT 20

Rows here distinguish A4 from A3: A3 is identity fragmentation regardless of source, A4 is specifically the bootstrap mismatch. A stale cached value and a flag edit between two page loads give the same rows: confirm with $feature_flag_bootstrapped_response on the event, and with the flag's history. If the query returns no rows and no events for the flag have $used_bootstrap_value = true anywhere, bootstrap is likely not in play and A4 is unlikely — but absence isn't definitive (older SDKs may not stamp the property). Cross-check by asking whether the user's app is server-rendered with bootstrapping enabled. For the second shape, the non-bootstrap response is often false: drop the variant filter from the query to see it, and query $feature_flag_called, because $experiment_exposure holds no false response.

Recommend:

  • Pass distinctID in the bootstrap payload when the server already knows the identity (e.g. logged-in users).
  • Bootstrap should be used together with server-side local evaluation, not alone.
  • Pass person properties to the evaluation in the type the release c