AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

debugging-experiments

Debug and support PostHog Experiments (A/B tests) for a customer looking at their own results. Use whenever an experiment support ticket is pasted or a customer asks a results question, most commonly "why aren't my exposures even?", "why is one variant getting no traffic?", "why am I missing / seeing too few exposures?", "why does the bias banner show?", or "why don't PostHog's numbers match my SQL?". Pulls the experiment's real data read-only, matches it to a known-cause catalog, and produces a customer-facing explanation, fix, and review of the pertinent numbers. Loads diagnosing-experiment-health as its deep diagnostic library. DO NOT TRIGGER when: creating an experiment (use creating-experiments), only configuring rollout (configuring-experiment-rollout) or metrics (configuring-experiment-analytics), asking lifecycle questions (managing-experiment-lifecycle), or the underlying feature flag is what's misbehaving rather than the results (use debugging-feature-flags).

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read posthog--debugging-experiments at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 4

references/pulling-the-data.md · 24000

first_day, countIf(variant = '<variant_a>') AS a, countIf(variant = '<variant_b>') AS b, round(countIf(variant = '<variant_a>') / greatest(countIf(variant = '<variant_b>'), 1), 3) AS a_over_b FROM first_exposures GROUP BY first_day ORDER BY first_day


_Pre-launch variant:_ rerun with `timestamp` bounded to the window **before** `start_date` (and
without the `IN (...)` filter, so `false`/`null` show). If the flag was live pre-launch and the
same directional skew is already present — before anyone saw the new UX — the cause is
**assignment**, not capture.

**First-exposure surface split (capture-by-surface).** Some paths ~50% and others ~100% one
variant ⇒ that arm reaches a surface the other can't. Swap `$pathname` for `$current_url` if too
coarse, or `$screen_name` for native apps.

```sql
WITH first_exposure AS (
    SELECT person_id,
           argMin(properties.$feature_flag_response, timestamp) AS variant,
           argMin(properties.$pathname, timestamp) AS first_path
    FROM events
    WHERE event = '$feature_flag_called'
      AND properties.$feature_flag = '<flag-key>'
      AND properties.$feature_flag_response IN ('<variant_a>', '<variant_b>')
      AND timestamp >= '<start_date>'
    GROUP BY person_id
)
SELECT coalesce(first_path, '(none)') AS path,
       countIf(variant = '<variant_a>') AS a,
       countIf(variant = '<variant_b>') AS b,
       count() AS total,
       round(countIf(variant = '<variant_a>') / count() * 100, 1) AS pct_a
FROM first_exposure
GROUP BY path
ORDER BY total DESC
LIMIT 40

Dropped false/null bucket by SDK. If a variant is short by ~N persons and the false/null person count is near N and concentrated on one $lib/surface, flag-read timing is the lead.

SELECT coalesce(toString(properties.$feature_flag_response), 'null') AS response,
       coalesce(properties.$lib, '(none)') AS lib,
       count() AS exposure_events,
       count(DISTINCT person_id) AS persons
FROM events
WHERE event = '$feature_flag_called'
  AND properties.$feature_flag = '<flag-key>'
  AND timestamp >= '<start_date>'
GROUP BY response, lib
ORDER BY exposure_events DESC
LIMIT 30

SDK split — a server SDK on a stale local-eval definition shows as a skewed server row ($lib = posthog-python/-node/-ruby/-go/-php) beside a clean web row. Group the first-exposure variant by $lib / $lib_version.

Bootstrap / local-eval mix per variant — group the exposure rows by properties.$used_bootstrap_value and properties.locally_evaluated. $used_bootstrap_value = true concentrated on the heavier arm is the signature of a bootstrap value inherited onto a fresh distinct_id instead of being hashed (an assignment-side cause; see bias-and-skew.md A4).

Query gotchas

  • Timezone. HogQL compares timestamp in the project timezone, not UTC. A launch bound written as UTC can be off by the project's offset (e.g. an hour on Europe/London) — verify the total exposures the query returns matches posthog:experiment-results-get before trusting any slice.
  • Property access. If the parser rejects properties.$feature_flag, use properties['$feature_flag'].
  • Escape every value you substitute into a placeholder. <flag-key>, <variant_a>, <start_date> and any distinct_id land inside single-quoted SQL literals, and posthog:execute-sql takes no bound parameters — so a value carrying a ' closes the literal early and the rest is parsed as SQL. distinct_ids are whatever the SDK sent, and variant keys are charset-validated only in the UI, so neither is safe to paste raw. HogQL escapes a quote as \' and a backslash as \\ inside a literal; apply that to the value before substituting. A distinct_id like x' OR 1=1 -- otherwise silently widens the predicate to every user in the project and you diagnose against the wrong rows.
  • Attributing edits. The posthog:advanced-activity-logs-list "feature flag updated" row does not carry the flag key — use posthog:feature-flags-activity-retrieve { id: <feature_flag_id> } for the field-level diff when you need to prove which flag changed.

5. Change history

  • posthog:feature-flags-activity-retrieve { id: <feature_flag_id> } — flag edits with field-level diffs. Most "why did the numbers change?" surprises are a variant/rollout/ condition change visible here.
  • posthog:advanced-activity-logs-list { scopes: ["Experiment"], item_ids: [<experiment_id>] } — experiment-level timeline (who/when; no change diff, so use it for when, not what).

Handing off

If a number disproves a cause, drop it; if it confirms one, lead the reply with that evidence. Convert every internal number to the customer's language before quoting it — see customer-reply.md.