AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

configuring-experiment-analytics

Configures the analytics side of a PostHog experiment — exposure criteria (server-resolved default exposure event vs custom exposure events), primary and secondary metrics, the supported metric types (count, sum, ratio with `math` and `math_property`, retention with `retention_window_start` and `start_handling`), multivariate user handling ("Exclude from analysis" vs "Use first seen variant"), and how to read results once the experiment is live. Use when the user adds or edits a primary or secondary metric (e.g. "add a secondary metric tracking 'downloaded_file' per user"), sets up a ratio metric (e.g. "revenue from purchase_completed / pageviews"), sets up a retention metric (e.g. "$pageview → uploaded_file, 7-day window"), configures custom exposure (e.g. "only count users who hit /checkout"), changes multivariate handling, or asks "who is in the analysis?", "how do I measure impact?", "is this winning?", "what's the confidence level?", or "should I ship?".

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--configuring-experiment-analytics at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 4

references/interpreting-results.md

Interpreting experiment results

Getting results

Use experiment-timeseries-results with the metric_uuid and fingerprint from the experiment's metrics array. Get the experiment first via experiment-get to find these values.

Statistical significance

  • Only recommend shipping when results are statistically significant
  • Bayesian experiments report probability of each variant being best
  • Frequentist experiments report p-values and confidence intervals

Do NOT recommend shipping just because a variant is "winning" — check significance first.

Sample size and runtime

  • Experiments typically need 1-2 weeks minimum for reliable results
  • Small sample sizes produce unreliable results — warn the user
  • If the experiment just launched, set expectations about when results will be meaningful

Multiple metrics

Each metric may tell a different story. Present the full picture:

  • Primary metric improved but secondary degraded? Call it out.
  • Some metrics significant, others not? Report honestly.
  • Don't cherry-pick the metric that supports shipping.

Decision framework

| Situation | Recommendation | | ----------------------------------------------------- | ------------------------------------------------------------ | | Clear winner, significant results, sufficient runtime | Ship the winning variant | | No significant difference after 2+ weeks | End as inconclusive — the variants don't meaningfully differ | | Primary improved but guardrail metric degraded | Flag the trade-off, let the user decide | | Results are borderline significant | Recommend continuing to run, or end as inconclusive | | Very early results (< 1 week) | Too early to draw conclusions — wait |

What NOT to do

  • Don't declare an experiment failed based on early results
  • Don't recommend shipping based on borderline significance
  • Don't ignore secondary/guardrail metrics when primary looks good
  • If results are ambiguous, say so — let the user decide