references/bias-and-skew.md · 48000
rning visitor gets the cached value.
- The application's own fallback
[MEDIUM]. Code that reports "no variant" ascontrolputs everyone outside the rollout into control, in every count that the application fills: the user's own analytics, and a custom exposure event that carries the application's value. Control there holds every person the flag gives no variant, on top of its own share of the people with a variant. The default exposure event is not affected: its response staysfalseand is no exposure. - A holdout.
People in a holdout get
holdout-<id>in place of a variant. That is intended: they are outside the experiment.
Evidence to gather.
- The responses of
$feature_flag_calledby day and by library ("By day and library" indiagnostic-snapshot.md, with$feature_flag_calledas the event): the share that is no variant key, and whether it sits in one library.$experiment_exposureholds nofalseor empty response, so it cannot show these rows. feature_flag.filters.groups[].properties: a condition on a property that is set after sign-up, or by the backend.feature_flag.bucketing_identifier.- A
falseshare that equals what the release conditions exclude by design is not a finding.
Co-occurs with: A2, A4, A9, and B10 in empty-experiment.md.
Recommendation, in order of preference:
- Send the exposure from the place that knows the variant the user got.
- Put the variant on the exposure event explicitly.
- Make the flag's inputs available before the evaluation: pass the person properties with the call, and wait for flags to load.
- Avoid release conditions on data that does not exist at evaluation time.