AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

debugging-experiments

Debug and support PostHog Experiments (A/B tests) for a customer looking at their own results. Use whenever an experiment support ticket is pasted or a customer asks a results question, most commonly "why aren't my exposures even?", "why is one variant getting no traffic?", "why am I missing / seeing too few exposures?", "why does the bias banner show?", or "why don't PostHog's numbers match my SQL?". Pulls the experiment's real data read-only, matches it to a known-cause catalog, and produces a customer-facing explanation, fix, and review of the pertinent numbers. Loads diagnosing-experiment-health as its deep diagnostic library. DO NOT TRIGGER when: creating an experiment (use creating-experiments), only configuring rollout (configuring-experiment-rollout) or metrics (configuring-experiment-analytics), asking lifecycle questions (managing-experiment-lifecycle), or the underlying feature flag is what's misbehaving rather than the results (use debugging-feature-flags).

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read posthog--debugging-experiments at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 4

references/pulling-the-data.md

Pulling the data

Run this sequence read-only before diagnosing or asking the customer anything. It produces the numbers you will cite back to them. The queries reuse the templates verified in ../../diagnosing-experiment-health/references/diagnostic-snapshot.md — read that file for the full rationale and edge cases; this is the customer-support-focused subset plus the numbers each cause needs.

1. Config — posthog:experiment-get

Pull these fields; they are inputs to almost every cause:

  • exposure_criteria.multiple_variant_handling — "exclude" (default) or "first_seen".
  • exposure_criteria.exposure_config.event — set means a custom exposure event (changes the query below); absent means the default $feature_flag_called.
  • exposure_criteria.filterTestAccounts — defaults to true.
  • feature_flag.filters.groups[] — per group read variant, properties, rollout_percentage (that group's rollout, % of the matched bucketing units — persons, or groups when aggregation_group_type_index is set — that enter the test; each group has its own, so there is no single overall rollout unless the flag has one unconditional group). Any non-null variant is a forced assignment (not randomized).
  • feature_flag.filters.multivariate.variants[] — the variant keys, their rollout_percentage (the configured split), and their stored order; the offline hash-recomputation test walks them in this order, so read it from the live flag.
  • feature_flag.ensure_experience_continuity — if true, assignment hashes a stored override key, so the offline hash test is unreliable (see the decisive test below).
  • feature_flag.filters.aggregation_group_type_index — if set, the experiment is group-aggregated (randomizes and counts groups, not persons); see the note below.
  • feature_flag.bucketing_identifier — if "device_id", the flag buckets on the device ID ($device_id), not distinct_id, so the offline recompute must hash $device_id (see the device-ID note below). Only applies to person-aggregated flags — group aggregation takes precedence. Absent / "distinct_id" is the default.
  • feature_flag.filters.holdout — if present, a global holdout deterministically excludes a slice of users from the experiment (see the holdout note below). Cross-check with posthog:experiment-holdouts-list.
  • Flag dependencies — a property of type flag inside feature_flag.filters.groups[].properties means this flag depends on another flag and fails closed when the parent isn't matched (see the dependency note below). List dependents with posthog:feature-flags-dependent-flags-retrieve.
  • feature_flag.active, status, start_date, end_date, stats_config.

Group-aggregated experiments. When aggregation_group_type_index is set, the flag buckets and counts groups (e.g. companies), not persons. Everywhere below that uses person_id as the unit, count the group key instead (the $group_<index> / $groups value the flag aggregates on), and the offline recompute in the decisive test hashes that group key rather than distinct_id. Counting persons on a group-aggregated experiment overstates N and can manufacture an SRM that isn't there.

Device-ID bucketing. When feature_flag.bucketing_identifier == "device_id", a person-aggregated flag hashes the device ID (the $device_id on the exposure event), not the distinct_id — so a single person keeps one variant across logins, but the same person on a second device can land in the other arm. The decisive test must therefore hash $device_id, not distinct_id: hashing distinct_id makes correctly-assigned users look like disagreements and misreads a capture-side SRM as assignment-side. Production falls back to distinct_id when $device_id is empty, so export coalesce(nullIf($device_id, ''), distinct_id) to mirror it. Group aggregation takes precedence over this (a group flag hashes the group key).

Holdouts. A global holdout deterministically excludes a slice of users (hashed separately with a holdout- prefix) from the experiment; those users are recorded with a holdout-<id> response, not control/test. Because the holdout hash is independent of the variant hash, it removes users evenly from both arms — so it lowers the analyzable N but does not create a directional SRM. When the customer says "fewer users than I expected," a holdout is a common benign answer: count the holdout-<id> bucket in the §3 breakdown against the shortfall before hunting for a bug.

Flag dependencies. A flag can gate on another flag via a property of type flag in its release conditions. Dependencies fail closed: a user who doesn't match the parent evaluates to false (no variant) rather than being randomized. So a dependency can both shrink a population (users drop to false) and skew it if the parent's own rollout correlates with anything. Detect with posthog:feature-flags-dependent-flags-retrieve or the type-flag property in filters.groups[]; the fix (widen/align the parent, or drop the dependency) is in the customer's flag config.

2. Metrics + exposure totals — posthog:experiment-results-get

Returns per-variant exposure totals and metric results in one call:

  • exposures.total_exposures[variant] — including the $multiple bucket (users exposed to more than one variant). $multiple share = total_exposures["$multiple"] / sum(all).
  • exposures.timeseries[] — daily exposure_counts per variant, for trajectory/flat-tail.
  • metrics.primary.results[] / metrics.secondary.results[] — each row carries index, a metric summary, and data (the primary/secondary object itself also has a count); a data: null row is failed-or-not-yet-computed, not necessarily broken. Re-pull, or force one recompute with posthog:experiment-results-get { refresh: true }, before reporting a metric as failing (see the transient-vs-real protocol in diagnostic-snapshot.md).

3. Exposure shape — posthog:execute-sql

Default exposure event:

SELECT
  properties.$feature_flag_response AS variant,
  count() AS exposure_events,
  count(DISTINCT person_id) AS persons,
  count(DISTINCT distinct_id) AS distinct_ids,
  min(timestamp) AS first_seen,
  max(timestamp) AS last_seen
FROM events
WHERE event = '$feature_flag_called'
  AND properties.$feature_flag = '<flag-key>'
  AND timestamp >= '<start_date>'
GROUP BY variant
ORDER BY exposure_events DESC

exposure_events is not the SRM input. It counts raw $feature_flag_called events, and one user fires as many as their app evaluates the flag — a variant that re-renders or adds a route reads it more often, so the event ratio drifts from the person ratio with nothing wrong. Use this column for volume and liveness (is anything arriving, has one arm gone quiet), and take the SRM counts from §2's total_exposures, which is already one row per person. See the chi-squared section in §4.

If exposure_criteria.exposure_config.event is set, the variant attribution lives on properties.$feature/<flag-key> instead of $feature_flag_response, and the event filter is the custom event — see diagnostic-snapshot.md for the custom-event variant of this query.

A holdout-<id> row (if the experiment has a holdout) is expected and correctly excluded from the control/test split — its count is the size of the holdout, useful for explaining a population shortfall (see the holdout note in §1).

Ignore $feature_flag_response = false / None / null rows: $feature_flag_called fires on every evaluation, including users who didn't bucket into the test. They aren't a balance signal (PostHog's own analysis filters them out). The exception is when every row is None/false — that's a missing-exposures symptom (a flag dependency failing closed lands users here too). Second exception — a capture-side SRM: when one arm is short, this bucket's volume (broken down by $lib/surface, and checked against the short-arm gap) can be the mechanism rather than noise — users who read the flag before it loaded get false/undefined and are dropped from their arm instead of showing up wrong. The SRM localization queries below quantify it.

4. The numbers each cause needs

| Cause | The number that confirms it | | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | Uneven split + Exclude bias | uneven rollout_percentage split and $multiple share > 0.1% (banner threshold), handling = exclude | | Sample ratio mismatch | chi-squared p < 0.001 on §2's per-person total_exposures (see below); only meaningful once totals are healthy | | Assignment override (assignment-side SRM) | the reassignment component carries the gap in the decisive test below; then localize with the SDK split / bootstrap mix below | | Capture-by-surface (capture-side SRM) | a $pathname/$screen_name where one variant's share jumps to ~100% while other paths sit near the split | | Flag read before load (capture-side SRM) | false/null person count near the short-arm gap and concentrated on that arm's $lib/surface | | Pre-launch skew | the same directional skew before start_date ⇒ points at assignment, not capture | | Bootstrap-inherited variant | $used_bootstrap_value = true concentrated on the heavier arm | | Identity fragmentation | distinct_ids / persons > ~1.2 for a variant, or persons under multiple variants | | No randomization / forced variant | a feature_flag.filters.groups[] entry with non-null variant + broad properties | | Mid-run rebucketing | residual exposures for a 0%-configured variant + a flag filters diff after start_date | | Missing exposures | total exposures ~0, or a variant last_seen days behind the other, or a flat timeseries tail | | Test-account exclusion | count of exposures matching the project's test-account filter | | Holdout population loss | size of the holdout-<id> bucket vs the expected-minus-actual N (removes users evenly, no directional SRM) | | Flag dependency (fail-closed) | false responses concentrated on users failing a type-flag release condition; posthog:feature-flags-dependent-flags-retrieve names the parent | | Capture disabled | exposures ~0 while the flag is demonstrably read — send_feature_flag_events/events-off in the SDK config, not a wrong accessor |

Sample ratio mismatch (SRM) — chi-squared

Count people, not events. The unit is one row per person at first exposure — take Oᵢ from §2's exposures.total_exposures[variant], which the product already collapses per person, not from §3's exposure_events. The test assumes each user is one independent draw from the split; raw $feature_flag_called counts break that, because events-per-user varies by arm. Running χ² on event counts inflates it and manufactures an SRM on a perfectly balanced experiment — which then sends the decisive test below to ~100% agreement and the diagnosis off hunting a capture-side surface split that was never there. On a group-aggregated experiment the unit is the group key, not the person (see §1).

Compare those observed counts to the configured split. For observed counts Oᵢ with total N and configured proportions pᵢ, expected Eᵢ = N·pᵢ, and χ² = Σ (Oᵢ − Eᵢ)² / Eᵢ with k − 1 degrees of freedom. Flag SRM only at p < 0.001. Don't call SRM below ~1,000 exposed persons/variant — small-sample variance dominates. Exclude the $multiple bucket from this test (it isn't a variant). For a two-variant split the equivalent z-test is z = (O₁ − E₁) / sqrt(N·p₁·(1−p₁)); |z| ≳ 6 is the ~4.7e-11 range and unmistakable. Multivariate (3+ arms): the chi-squared test above already generalizes (k − 1 d.o.f.), and srm_check.py and every localization query below take N variant keys — the two-arm z-test is just shorthand. With 3+ arms, read which arm carries the χ² by comparing each (Oᵢ − Eᵢ)² / Eᵢ term, then localize that arm.

Back-check the split you're testing against. PostHog's SRM check reads the expected proportions from the live flag's multivariate.variants, not a stored copy on the experiment. If in doubt, ask which configured split reproduces the observed p — only one will (e.g. 832 vs 1,123 gives p ≈ 4.7e-11 under 50/50, but p ≈ 0.03 under 45/55 and no SRM at all under 43/57). Matching the p pins down the split the customer is actually running.

Localizing a confirmed SRM (assignment vs capture)

Run these only after SRM is confirmed. First run the decisive test below to pick the half, then run the matching queries: a capture verdict sends you to the surface / dropped-false / SDK queries, an assignment verdict to the SDK / bootstrap queries. All queries collapse to one row per person at first exposure, mirroring the product; substitute the two real variant keys. They assume the default exposure event — for a custom exposure event, swap the event filter and read the variant from properties.$feature/<flag-key> instead (as in §3), and for a group-aggregated flag count the group key rather than person_id (see §1).

The decisive test: recompute assignment offline

Recompute each user's variant from the flag hash, then split the gap between the recorded split and the configured split into the only two places it can come from. For each variant, over a sample of n identifiers, with expected = n × configured share:

recorded − expected  =  (predicted − expected)  +  (recorded − predicted)
  the observed gap        selection component       reassignment component
                          ⇒ CAPTURE-side            ⇒ ASSIGNMENT-side

That's an identity, not a heuristic — the two components always sum to the gap. predicted is the hash-recomputed variant, so:

  • Selection carries the gap ⇒ the users who got recorded were already a skewed draw before assignment is even considered — capture-side. Localize with the surface / dropped-false / SDK queries below.
  • Reassignment carries the gap ⇒ something overrode assignment at serve time — assignment-side. Localize with the SDK / bootstrap queries below; suspect stale local-eval, an inherited bootstrap value, a forced release-condition variant, or a mid-run rehash.

Don't route on the raw agreement percentage. A handful of scattered disagreements can't produce a directional SRM, so agreement alone doesn't separate the halves: a large capture-side skew sitting under ~2% of unrelated override noise reads as "98% agreement" and looks assignment-side, when selection is carrying the entire gap. The share of the gap each component explains is the number that routes; srm_check.py prints both, each with its own significance test, and refuses to name a side when neither dominates.

The algorithm — verified byte-exact against PostHog's implementation in rust/feature-flags/src/flags/flag_matching.rs (get_matching_variant) and flag_matching_utils.rs (calculate_hash). Don't eyeball it — the salt is easy to get wrong:

  1. hash_key = f"{flag_key}.{identifier}variant" — note the . after the key and the literal variant salt. (The plain rollout gate hashes with an empty salt; the variant walk uses "variant". Using the wrong salt is the classic reimplementation bug.) identifier is the distinct_id by default — the group key when the flag is group-aggregated, or the device ID ($device_id) when feature_flag.bucketing_identifier == "device_id" (see §1). Hashing the wrong one inverts the verdict.
  2. h = int(sha1(hash_key).hexdigest()[:15], 16) / 0xfffffffffffffff — a float in [0, 1).
  3. Walk filters.multivariate.variants in stored order, accumulating rollout_percentage / 100 into a cumulative bound; the first variant with h < cumulative is the assigned variant.

Don't hand-run this — the bundled srm_check.py implements it. --selftest replays the repo's golden hash vectors first, so you confirm the reimplementation matches this build before trusting a verdict:

Save the flag's filters.multivariate.variants array to a file and pass the path — that preserves stored order and keeps the variant keys out of the shell:

./srm_check.py --selftest
./srm_check.py --flag-key <flag-key> --variants-file variants.json --csv exposures.csv

Never interpolate variant keys into the command line. Variant keys are charset-validated in the PostHog UI but not by the API, so a key on a flag that reached you through a ticket can contain shell metacharacters or quote characters. --variants-file reads them as JSON, so they stay data. The --variants control=50,test=50 form is a convenience for keys you have already read and eyeballed; don't build it by substituting values you haven't looked at.

Constraints:

  • Not pure SQL. SHA1 isn't in HogQL's whitelist, so this runs outside the database. Export the sample below and feed it to the script; a deterministic cityHash64 sample of 800 gives SE ~1.8pp, enough to separate 50/50 from 57/43. distinct_ids are often emails — keep them customer-side.
  • Order- and split-sensitive. The walk depends on the exact array order and percentages in the live flag's filters.multivariate.variants; a wrong order silently inverts the prediction. Pass them to --variants in stored order.
  • Population must match the SRM. The SRM is computed per person with the $multiple bucket excluded, so the export restricts to that same analyzable population — otherwise the agreement rate describes a different set of users than the gap you're explaining. Under first_seen handling no one is excluded, so drop the IN (SELECT ...) filter. The excluded $multiple persons are a finding in their own right, not a rounding error: size them from §2's total_exposures, since a large bucket on an uneven split is the bias-banner cause.
  • Ambiguous identifiers are evidence, not noise. variants_seen > 1 marks an identifier that recorded more than one variant over time — the mid-run-rehash and bootstrap-inheritance signature. Collapsing it with argMin would keep the earliest row and hide the disagreement entirely, so the export carries the count and the script reports it separately instead of averaging it away.
  • Per-identifier weighting. One person with several distinct_ids contributes several rows, so fragmented persons are upweighted relative to the per-person SRM. Cross-check against the distinct_ids / persons ratio from §3 — well above 1 means the agreement rate is identifier-weighted and the sample is not a clean stand-in for the person-level population.
  • Identifier must match production. The recompute is only valid if you hash the identifier production hashed — the group key for group flags, $device_id for device-bucketed flags (bucketing_identifier == "device_id"), otherwise distinct_id (see §1). Hash the wrong one and correctly-assigned users read as disagreements, so a clean assignment looks assignment-side. The script guards the worst case: agreement that can't beat the rate a coin flip would reach on this split (50% on 50/50) means the recompute carries no signal at all, and it reports the test as inapplicable rather than blaming assignment. A subtler mismatch still slips through — sanity guard: if agreement is far below 100% everywhere — including a slice you already know is balanced, or the pre-launch window — suspect a wrong identifier (or continuity, below) before concluding assignment-side.
  • Experience continuity. If ensure_experience_continuity = true, assignment hashes a stored override key you can't reconstruct from the identifier, so this test is unreliable — skip it and lean on the capture-side checks plus the activity log.
-- Deterministic sample for the offline recompute (n=800 → SE ~1.8pp; drop LIMIT for all rows).
-- One row per hashed identifier, restricted to the SRM-analyzable population so the agreement
-- rate describes the same users the SRM gap is computed over.
-- Custom exposure event: filter that event and read the variant from properties.$feature/<flag-key>.
-- Group-aggregated flag: export the group key instead of distinct_id, and group person_variant by it.
-- Device-ID bucketing (bucketing_identifier == "device_id"): select
--   coalesce(nullIf(properties.$device_id, ''), distinct_id) AS distinct_id instead — production
--   hashed the device id, so hashing distinct_id would fabricate disagreements.
WITH person_variants AS (
    SELECT person_id,
           -- Mirrors multiple_variant_handling = 'exclude' (the default).
           -- For 'first_seen': argMin(properties.$feature_flag_response, timestamp)
           if(uniqExact(properties.$feature_flag_response) > 1, '$multiple',
              any(properties.$feature_flag_response)) AS variant
    FROM events
    WHERE event = '$feature_flag_called'
      AND properties.$feature_flag = '<flag-key>'
      AND properties.$feature_flag_response IN ('<variant_a>', '<variant_b>')
      AND timestamp >= '<start_date>'
    GROUP BY person_id
)
SELECT distinct_id,
       argMin(properties.$feature_flag_response, timestamp) AS recorded_variant,
       uniqExact(properties.$feature_flag_response) AS variants_seen
FROM events
WHERE event = '$feature_flag_called'
  AND properties.$feature_flag = '<flag-key>'
  AND properties.$feature_flag_response IN ('<variant_a>', '<variant_b>')
  AND timestamp >= '<start_date>'
  -- Drop this filter under 'first seen' handling, which excludes no one.
  AND person_id IN (SELECT person_id FROM person_variants WHERE variant != '$multiple')
GROUP BY distinct_id
ORDER BY cityHash64(distinct_id)
LIMIT 800

Localization queries

Daily first-exposure ratio — flat = standing/structural bias; a step change on one day = a change made then (cross-check posthog:feature-flags-activity-retrieve for that date).

WITH first_exposures AS (
    SELECT person_id,
           if(uniqExact(properties.$feature_flag_response) > 1, '$multiple',
              any(properties.$feature_flag_response)) AS variant,
           toDate(min(timestamp)) AS first_day
    FROM events
    WHERE event = '$feature_flag_called'
      AND properties.$feature_flag = '<flag-key>'
      AND properties.$feature_flag_response IN ('<variant_a>', '<variant_b>')
      AND timestamp >= '<start_date>'
    GROUP BY person_id
)
SELECT