references/interpreting-results.md
Interpreting experiment results
Getting results
Use experiment-timeseries-results with the metric_uuid and fingerprint from the experiment's metrics array. Get the experiment first via experiment-get to find these values.
Statistical significance
- Only recommend shipping when results are statistically significant
- Bayesian experiments report probability of each variant being best
- Frequentist experiments report p-values and confidence intervals
Do NOT recommend shipping just because a variant is "winning" — check significance first.
Sample size and runtime
- Experiments typically need 1-2 weeks minimum for reliable results
- Small sample sizes produce unreliable results — warn the user
- If the experiment just launched, set expectations about when results will be meaningful
Multiple metrics
Each metric may tell a different story. Present the full picture:
- Primary metric improved but secondary degraded? Call it out.
- Some metrics significant, others not? Report honestly.
- Don't cherry-pick the metric that supports shipping.
Decision framework
| Situation | Recommendation | | ----------------------------------------------------- | ------------------------------------------------------------ | | Clear winner, significant results, sufficient runtime | Ship the winning variant | | No significant difference after 2+ weeks | End as inconclusive — the variants don't meaningfully differ | | Primary improved but guardrail metric degraded | Flag the trade-off, let the user decide | | Results are borderline significant | Recommend continuing to run, or end as inconclusive | | Very early results (< 1 week) | Too early to draw conclusions — wait |
What NOT to do
- Don't declare an experiment failed based on early results
- Don't recommend shipping based on borderline significance
- Don't ignore secondary/guardrail metrics when primary looks good
- If results are ambiguous, say so — let the user decide