AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

experiment-designer

Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--experiment-designer at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 3

references/experiment-playbook.md

Experiment Playbook

Experiment Types

A/B Test

  • Compare one control versus one variant.
  • Best for high-confidence directional decisions.

Multivariate Test

  • Test combinations of multiple factors.
  • Useful for interaction effects, requires larger traffic.

Holdout Test

  • Keep a percentage unexposed to intervention.
  • Useful for measuring incremental lift over broader changes.

Metric Design

Primary Metric

  • One metric that decides ship/no-ship.
  • Must align with user value and business objective.

Guardrail Metrics

  • Prevent local optimization damage.
  • Examples: error rate, latency, churn proxy, support contacts.

Diagnostic Metrics

  • Explain why change happened.
  • Do not use as decision gate unless pre-specified.

Stopping Rules

Define before launch:

  • Fixed sample size per group
  • Minimum run duration (to capture weekday/weekend behavior)
  • Guardrail breach thresholds (pause criteria)

Avoid:

  • Continuous peeking with fixed-horizon inference
  • Changing success metric mid-test
  • Retroactive segmentation without correction

Novelty and Primacy Effects

  • Novelty effect: short-term spike due to newness, not durable value.
  • Primacy effect: early exposure creates bias in user behavior.

Mitigation:

  • Run long enough for behavior stabilization.
  • Check returning users and delayed cohorts separately.
  • Re-run key tests when stakes are high.

Pre-Launch Checklist

  • [ ] Hypothesis complete (If/Then/Because)
  • [ ] Metric definitions frozen
  • [ ] Instrumentation validated
  • [ ] Randomization and assignment verified
  • [ ] Sample size and duration approved
  • [ ] Rollback plan documented

Post-Test Readout Template

  1. Hypothesis and scope
  2. Experiment setup and quality checks
  3. Primary metric effect size + confidence interval
  4. Guardrail status
  5. Segment-level observations (pre-registered only)
  6. Decision: ship, iterate, or reject
  7. Follow-up experiments