references/bias-and-skew.md
Bias & skew on a running experiment
Variant counts don't match the configured split, one variant looks biased, the in-app warning banner appeared, or users are showing up under multiple variants.
Before diagnosing
Read three values first.
The first two come from the exposure data of the Step 1.5 snapshot (exposures in experiment-results-get), the third from experiment-get.
experiment-results-get runs queries: call it once, after the stored results and the change history of the snapshot, and reuse its exposures for every check in this file.
The first two are computed on the population the experiment analyzes, which a query of your own does not reproduce.
- The sample ratio test:
exposures.sample_ratio_mismatch.p_value. The page flags a mismatch below 0.001.- Below 0.001: a real assignment or capture problem, or a split that changed during the run, because the test compares the whole run with the current split. Walk A2.
- At or above 0.001: no mismatch by the product's rule.
On a small sample the visible split is noise (see C2 in
interpretation.md). - A value between 0.001 and 0.05 is not cleared by that rule
[MEDIUM]. A slow leak of exposures in one arm looks like this for days before it crosses the threshold. No tool returns the p-value of an earlier day. Read the series inexposures.timeseries, which is cumulative per variant: take the difference between days, and walk A9 when the gap grows in one direction.
- People in more than one variant:
exposures.total_exposures['$multiple']over the sum of all values. Non-zero puts A1, A3 and A4 on the table, and A6 and A8 when the flag or the identifier changed during the run. Underfirst_seenhandling the key is absent, because each person keeps the first variant. - The configured split.
feature_flag.filters.multivariate.variants[].rollout_percentagefromexperiment-get. An uneven split amplifies whichever bias source is present. The flag shows the split of today. On an experiment that has ended, or whose flag was edited, take the split of the run from the change history ("Changes after launch" indiagnostic-snapshot.md).
When the observed split matches the split the flag had during the run, there is no mismatch to explain.
A user who expects 50/50 on a flag that ran at 90/10 has a question about the configuration, not about bias: say which split the flag had and since when.
If the split changed after launch, walk E3 in mid-run-changes.md.
The product's sample ratio test compares against the flag's current split, so after such a change its p-value tests the wrong expectation.
If the symptom is "metric count is far smaller than exposures" (e.g. 10× or 100× gap), walk this
file before numbers-vs-sql.md — that shape of divergence is most often a bucketing / identity
problem (A3/A4), not a query-scope problem.
The SQL in this file follows the query rules in diagnostic-snapshot.md: the experiment's exposure event, the flag's variant keys, a window with both ends set, and no test-account filter.
The exposure event is resolved_exposure_event, unless exposure_criteria.exposure_config names $experiment_exposure: then it is that event ("Which event, which property" in diagnostic-snapshot.md).
When resolved_exposure_event is $feature_flag_called, also read $experiment_exposure (event IN ('$feature_flag_called', '$experiment_exposure')), and count persons, not rows.
The queries count person_id.
On a flag aggregated by a group type (feature_flag.filters.aggregation_group_type_index is set), put $group_<index> in its place and leave out the rows where it is empty.
When a result is compared with $multiple, leave out the keys in excluded_variants: the experiment never counts them.
resolved_exposure_event is always the default event.
On an experiment with a custom exposure event these queries read the flag's calls, not the people the experiment counts: say so when you report them.
Contents
- A1 — Multi-variant exclusion bias on uneven split
- A2 — Sample ratio mismatch (SRM)
- A3 — Identity fragmentation (users in both control and test)
- A4 — Bootstrap ×
/flagsvariant disagreement - A5 — Flag/experiment state inconsistency
- A6 — Mid-run flag edits that rebucket already-exposed users
- A7 — Non-randomized assignment via release conditions
- A7b — A forced-variant group starves the other arm
- A8 — Migrating the
distinct_idstrategy during a running experiment - A9 — Exposure events are lost or sent unevenly between the arms
- A10 — Part of the audience gets no variant from the flag
A1 — Multi-variant exclusion bias on uneven split [HIGH]
This is the signal behind the in-app warning "The uneven split may bias the results".
Evidence. The banner's check reports a finding when all four hold:
multiple_variant_handling == "exclude"(the default)- the variant rollouts are uneven. A variant at 0% counts, so a flag at 0/50/50 is uneven.
- the share of people in more than one variant is above 0.1%.
The value is
exposures.bias_risk.multiple_variant_percentage: the$multiplecount over all exposures. - the experiment has no end date. After the end the check reports nothing, and the banner is gone. The exclusion is still in the numbers: the banner's absence on an ended experiment clears nothing.
The warning-vs-visible gap. The banner fires above 0.1%.
The collapsed summary of the Exposures panel leaves the $multiple row out at or below 0.5%.
The expanded table always lists it, labeled "(excluded from analysis)".
So a user can see the banner while the summary shows a clean split.
The warning is real: point the user to the expanded table, and state the exact share from exposures.bias_risk.
Mechanism. Multi-variant users are dropped, but the smaller variant loses a larger fraction of its assignments than the larger variant. Multi-device / multi-session / signup-flow users tend to be high-intent — so the smaller variant keeps a low-intent slice and looks worse than it should. This is asymmetric exclusion bias, not a UI bug.
Recommend (in this order):
- Switch to an equal split. See
configuring-experiment-rollout. On a draft experiment this is free. Mid-run it's an anti-pattern — prefer reset or end+restart over changing the split mid-run. The banner's own "Adjust distribution" button opens the split editor on the running experiment. A user who followed it changed the split mid-run: read the flag's history before any other cause (E3 inmid-run-changes.md). - Switch
multiple_variant_handlingto"first_seen". Seeconfiguring-experiment-analytics. Mid-run this is the low-disruption option — no users switch variants, all already-collected data stays in the analysis.first_seenis less biased thanexcludefor this specific shape, not unbiased: it counts the first variant a user saw and ignores later ones, which still asymmetrically discounts engaged multi-session users. There is no clean fix for the underlying problem; the trade-off is between which bias the user prefers.
The experiment's settings and the public docs call Exclude the recommended handling. That holds for an even split, and a user may quote it: the banner is the product's own exception for uneven splits.
A2 — Sample ratio mismatch (SRM) [HIGH]
Evidence. exposures.sample_ratio_mismatch holds the p-value of a chi-squared test and the expected count per variant.
- The test runs once the variants with a rollout above 0% hold 100 exposures together.
- With fewer than two variants above 0% there is no test, and the value is null.
That is the state after a ship and after a split edited to 100/0.
Judge such an experiment by the split of the run ("Changes after launch" in
diagnostic-snapshot.md). - The page flags a mismatch at p < 0.001. The server returns the p-value and applies no threshold.
- The expected counts follow the flag's split as it is now.
They are rescaled for a holdout.
$multiple, the holdout and excluded variants are left out. - After a split change the test compares the whole run with the new split, so a mismatch is the expected reading. Read A6 before anything else in that case.
MUST read the product's value. MUST NOT run the test on event counts of your own: raw exposure events include test accounts, responses that are no variant, repeated events of one person and events outside the analysis window.
What it means. The distribution of exposed people differs from the flag's current split by more than chance allows. Either the split changed during the run (cause 1), or something biases the assignment or the capture of exposures. A visible gap on a small sample is not a mismatch until the test says so, and a mismatch on a small sample is still a mismatch: the test already accounts for the sample size. The exception is a single reading that clears on the next day's data, which happens about once in a thousand readings by construction.
Causes, cheapest evidence first.
Surface each cause the evidence supports (Step 3 of SKILL.md).
-
The flag changed after launch. Evidence: a change of the flag's
filtersafterstart_dateinexperiment-activity. Which edits move people, and which do not: A6. -
Exposure events are lost or sent unevenly between the arms
[MEDIUM]. The flag assigns evenly and the event does not arrive evenly: a redirect, an exposure placed after the arms diverge, a flag read before flags load. It is the most frequent cause of a mismatch that no flag change explains. See A9. -
Part of the audience gets no variant. Responses of
falseor an empty value are no exposures, and the people who drop out are not a random share. The flag decides whether a person gets a variant apart from which variant, so afalsefrom the flag removes people from every arm alike. It moves the split only when something ties the loss to one arm, such as the application's own fallback on a custom exposure event, or a pinned condition that still matches without a device id (A10). See A10. -
A release condition pins a variant. Evidence: a
variantinfeature_flag.filters.groups[]that is one of the flag's variant keys. See A7. -
Server and client evaluate the flag differently
[MEDIUM]. A bootstrapped value against a later evaluation (A4). A server that evaluates locally keeps the old definition for its polling interval after a flag edit. A flag with persistence across authentication is not evaluated locally: the SDK falls back to a remote call, so the two paths can answer at different times with different inputs. WithstrictLocalEvaluationoronlyEvaluateLocallyset in posthog-node there is no remote call, and the call returns no value, which is no exposure. Evidence: exposures per variant by library (snapshot query),feature_flag.ensure_experience_continuity, and whether the user's servers evaluate locally. -
Bots on server-side evaluation
[LOW]. The public docs rank this first. As a cause of a directional mismatch it is weak: one crawler is one identifier and counts once. It matters when a crawler sends a new identifier with every request. Evidence: the share of server-side libraries among the exposures, and exposed people with no later event. Fix path: the "Filter Bot Events" transformation (a data pipeline transformation) applies its built-in bot lists to browser events only. For events from a server SDK it filters only what the user adds as custom user-agent patterns or IP prefixes. With "Keep events where the useragent is not set?" at No, it also drops every event without the user-agent property ($raw_user_agentby default), and posthog-node sets that property only when the code passes it. The more reliable fix is to not evaluate the flag for requests the server already knows to be bots.
What does not cause a mismatch.
- Repeated exposure events of one person. The experiment counts each person once, at the first exposure. A server SDK that sends the event again after a restart or after its dedup cache fills up changes the event count, not the split.
- Ad-blockers and network loss. Both arms lose alike. The sample shrinks, and a reverse proxy recovers part of it.
- A holdout. The expected counts are rescaled for it.
- People in more than one variant, on an even split. They leave both arms alike. On an uneven split see A1.
Fix paths change the flag.
An agent can edit a flag over MCP.
On a running experiment every such edit is a mid-run change (A6, and Step 4 of SKILL.md): explain the effect and get the user's decision before any edit.
A3 — Identity fragmentation (users in both control and test) [MEDIUM]
This is an identity problem, not a bias problem. One human has two or more distinct ids. Each id is hashed by itself, so each can land in a different variant.
- When the ids are linked to one person, the analysis sees one person with two variants and counts the person as
$multiple. - When the ids were never linked, the analysis sees two persons with one variant each. Nothing in the exposure data shows it.
Evidence.
-
The count.
exposures.total_exposures['$multiple']. No event carries$multiple. The analysis computes it per person, and only from the flag's current variant keys: a response offalse, an empty response or an old key never makes a person$multiple. -
Under
first_seenhandling the count is not reported. The query below is then the only source. -
Ids per person. In the raw-responses query of the snapshot,
distinct_ids / personsabove 1 for a variant means that people evaluated the flag under several linked ids. That is the precondition for a flip. It says nothing about ids that were never linked. -
The people, when the cause has to be named. Start with one day on a large project. The query finds only people with two variants inside its window, so an empty result on a short window does not rule out A3:
-- People whose exposure events carry more than one variant key SELECT person_id, uniq(properties.$feature_flag_response) AS variants_seen, uniq(distinct_id) AS distinct_ids, groupUniqArray(properties.$feature_flag_response) AS variants FROM events WHERE event = '<resolved_exposure_event>' AND properties.$feature_flag = '<flag-key>' AND properties.$feature_flag_response IN ('<variant-key-1>', '<variant-key-2>') AND timestamp >= toDateTime('<window_start>', 'UTC') AND timestamp < toDateTime('<window_end>', 'UTC') GROUP BY person_id HAVING variants_seen > 1 ORDER BY variants_seen DESC, distinct_ids DESC LIMIT 20 -
One person's exposure events, to tell the causes apart:
SELECT timestamp, distinct_id, properties.$feature_flag_response AS response, properties.$lib AS lib FROM events WHERE event = '<resolved_exposure_event>' AND properties.$feature_flag = '<flag-key>' AND person_id = '<person_id>' AND timestamp >= toDateTime('<window_start>', 'UTC') AND timestamp < toDateTime('<window_end>', 'UTC') ORDER BY timestamp LIMIT 50- Different
distinct_ids: an identity cause from the list below. - The same
distinct_id, and a flag edit between the two timestamps: A6. - The same
distinct_id, no edit, different libraries: server and client disagree (A4, A10).
A response of
falsehas no$experiment_exposurerow: whenresolved_exposure_eventis$experiment_exposure, run the query on$feature_flag_calledto see a step to no variant. - Different
Common causes:
reset()was called between sessions (other than on logout)identify()ran after the flag was already evaluated- Cross-device usage without identity stitching
- Cookies cleared between visits, incognito / stealth browsing
- Anonymous → identified transition without flag persistence enabled
- The same user has different anonymous IDs client-side vs server-side, so the flag hash bucket differs
- Native mobile auth flows where the flag is read before the SDK identifies the user, or where authentication crosses an SDK boundary (e.g. web → in-app webview)
A note on what's fundamentally fixable vs not. Stitched-identity issues from identify()
ordering, cross-domain cookies, and bootstrap timing are real bugs that can be fixed. Multi-device
usage and incognito / stealth browsing are not fixable from PostHog's side — and the users who
exhibit them tend to be more engaged on average, so excluding the $multiple bucket pulls a
non-random slice out of the analysis. There is no clean fix; the recommendation is to contain the
problem (scope exposure to the relevant flow so the denominator stays meaningful) rather than
eliminate it.
Recommend:
- Audit
identify()andreset()ordering —reset()only on explicit logout,identify()before flag evaluation. Persistence across authentication carries a variant from anonymous to identified, not back: afterreset()the next evaluation is a new draw. - For experiments spanning logged-out → logged-in flows, consider one of:
- Persist flag across authentication steps (tradeoffs: requires
person_profiles: 'always', incompatible with local evaluation and bootstrapping, adds slight latency) - Device-ID bucketing — appropriate for landing/marketing/anonymous flows. Keeps the variant
stable across the anonymous→identified transition without the flag-persistence tradeoffs. Many
users don't realize this option exists; surface it explicitly when the symptom is cross-auth
bucketing.
Two limits.
An evaluation without a device id gets no variant (the flag returns
false), which is the usual case for a server-side call that passes none (A10). The exception is a release condition at a rollout of 100% that pins a variant: it still matches, and returns the pinned variant. And it cannot be combined with persistence across authentication: the API refuses the pair.
- Persist flag across authentication steps (tradeoffs: requires
- For pre-auth experiments, ensure cookies/localStorage persistence is configured (cookies preferred for cross-subdomain).
- For mobile flows, consider evaluating the flag server-side (local evaluation) once the user is authenticated rather than on first app open.
A4 — Bootstrap × /flags variant disagreement [MEDIUM]
Specific scenario: server-rendered app with bootstrapping enabled. The $multiple share in this
shape can become substantial — well above the trickle you'd expect from normal cross-device traffic
alone. Website-only flags (no bootstrap) are unaffected.
Mechanism. Two shapes.
- Different identifiers. The server bootstraps flags using the server-known
distinct_id, but the bootstrap payload doesn't includedistinctID— soposthog-jsinitializes with whatever's in persistence (often the anonymous ID). The bootstrap value gets reported under the anonymous ID; thenposthog-jsasks the flag service with whatever ID it has afteridentify(). When the IDs differ, the hashes differ, and one person sends exposure events for two variants. On a flag that buckets on the device id the identifier does not change atidentify(), so this shape needs different device ids. - Different inputs
[MEDIUM]. The bootstrap values come from the user's own backend, which evaluates the flag with the properties it passes. A property in another type or format than the stored one (a timestamp as a string against a stored number), or a variant set by hand, matches on one side only. The bootstrap-sourced exposure carries a variant, and the next evaluation returns another variant orfalse. The user sees the new experience for a moment and then loses it.
Evidence. The exposure event carries two source properties:
$used_bootstrap_value—truewhen the value was read before the flags request of that page load answered. That is a bootstrap value, and also a value cached from an earlier page load.$feature_flag_bootstrapped_responseholds what the bootstrap payload said for the flag, and is null without one.locally_evaluated—truewhen the event came from server-side local evaluation.
The signature is one person with an event read before the flags request answered and another event for the same flag, with different variants.
$used_bootstrap_value is usually not a materialized column, so this query reads the full properties of every row: keep the window to one day.
The comparison with true needs the property typed Boolean in the project, which is the usual case.
If the query fails on a type error, write the two conditions as toString(properties.$used_bootstrap_value) = 'true' and ifNull(toString(properties.$used_bootstrap_value), '') != 'true'.
SELECT
person_id,
groupUniqArrayIf(properties.$feature_flag_response, properties.$used_bootstrap_value = true) AS bootstrap_variants,
groupUniqArrayIf(properties.$feature_flag_response, ifNull(properties.$used_bootstrap_value, false) = false) AS other_variants
FROM events
WHERE event = '<resolved_exposure_event>'
AND properties.$feature_flag = '<flag-key>'
AND properties.$feature_flag_response IN ('<variant-key-1>', '<variant-key-2>')
AND timestamp >= toDateTime('<window_start>', 'UTC')
AND timestamp < toDateTime('<window_end>', 'UTC')
GROUP BY person_id
HAVING length(bootstrap_variants) > 0
AND length(other_variants) > 0
AND arraySort(bootstrap_variants) != arraySort(other_variants)
LIMIT 20
Rows here distinguish A4 from A3: A3 is identity fragmentation regardless of
source, A4 is specifically the bootstrap mismatch.
A stale cached value and a flag edit between two page loads give the same rows: confirm with $feature_flag_bootstrapped_response on the event, and with the flag's history.
If the query returns no rows and
no events for the flag have $used_bootstrap_value = true anywhere, bootstrap is likely not in
play and A4 is unlikely — but absence isn't definitive (older SDKs may not stamp the property).
Cross-check by asking whether the user's app is server-rendered with bootstrapping enabled.
For the second shape, the non-bootstrap response is often false: drop the variant filter from the query to see it, and query $feature_flag_called, because $experiment_exposure holds no false response.
Recommend:
- Pass
distinctIDin the bootstrap payload when the server already knows the identity (e.g. logged-in users). - Bootstrap should be used together with server-side local evaluation, not alone.
- Pass person properties to the evaluation in the type the release c