AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

experiment-designer

Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--experiment-designer at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 3

references/statistics-reference.md

Statistics Reference for Product Managers

p-value

The p-value is the probability of observing data at least as extreme as yours if there were no true effect.

  • Small p-value means data is less consistent with "no effect".
  • It does not tell you the probability that the variant is best.

Confidence Interval (CI)

A CI gives a plausible range for the true effect size.

  • Narrow interval: more precise estimate.
  • Wide interval: uncertain estimate.
  • If CI includes zero (or no-effect), directional confidence is weak.

Minimum Detectable Effect (MDE)

The smallest effect worth detecting.

  • Set MDE by business value threshold, not wishful optimism.
  • Smaller MDE requires larger sample size.

Statistical Power

Power is the probability of detecting a true effect of at least MDE.

  • Common target: 80% (0.8)
  • Higher power increases sample requirements.

Type I and Type II Errors

  • Type I (false positive): claim effect when none exists (controlled by alpha).
  • Type II (false negative): miss a real effect (controlled by power).

Practical Significance

An effect can be statistically significant but too small to matter.

Always ask:

  • Does the effect clear implementation cost?
  • Does it move strategic KPIs materially?

Power Analysis Inputs

For conversion experiments (two proportions):

  • Baseline conversion rate
  • MDE (absolute points or relative uplift)
  • Alpha (e.g., 0.05)
  • Power (e.g., 0.8)

Output:

  • Required sample size per variant
  • Total sample size
  • Approximate runtime based on traffic volume