assets/experiment_template.md
Chaos Experiment
Fill in every section before running. Refuse to run if any section is empty.
Identity
- Experiment ID:
<auto-generated; format: chaos-<target>-<attack>-<unix-ts>> - Date:
<YYYY-MM-DD> - Owner:
<your-handle@team> - On-call team:
<team channel / pager> - Reviewer:
<peer who reviewed this plan>
1. Hypothesis
When
<fault>,<steady-state metric>stays<tolerance>.
Example: When payment-svc is +200ms slow, checkout p99 stays below 500ms.
2. Steady-state metric
- Metric:
<e.g., p99 checkout latency> - Baseline window:
<e.g., 5 minutes pre-experiment> - Tolerance:
<e.g., within ±5% of baseline> - Dashboard:
<URL>
3. Attack
- Type:
[ ] latency [ ] error [ ] cpu [ ] memory [ ] disk [ ] network-partition [ ] dependency-failure [ ] time-skew [ ] kill-instance - Magnitude:
<e.g., +200ms> - Duration:
<minutes> - Target:
<service / pod / instance / region> - Tooling:
<Chaos Toolkit / Chaos Mesh / Litmus / Gremlin / AWS FIS / Custom>
4. Blast radius
- Traffic share:
<e.g., 5% of US> - Expected affected users:
<from blast_radius_calculator.py> - Error budget consumed:
<from blast_radius_calculator.py> - Risk score:
[ ] GREEN [ ] YELLOW [ ] RED
5. Abort criteria
Auto-trigger experiment termination if ANY of these hit.
- [ ]
<signal 1, e.g., p99 > 1000ms> - [ ]
<signal 2, e.g., 5xx rate > baseline + 1pp> - [ ]
<signal 3, e.g., on-call paged SEV1/SEV2>
6. Rollback procedure
<step to disable fault, e.g., "kubectl delete chaos networkchaos/<name>">- Verify steady state recovers within 2 minutes
- If not recovering, escalate as incident; restore from backup if needed
7. Learning question
What do you expect NOT to learn? Force yourself to predict.
<your prediction>
Pre-flight checklist
- [ ] Hypothesis written
- [ ] Steady-state metric measured for ≥5 min
- [ ] Blast radius calculated (GREEN or YELLOW only)
- [ ] Abort criteria documented with thresholds
- [ ] Rollback procedure tested in staging
- [ ] On-call team notified
- [ ] Monitoring dashboards open
- [ ] Owner identified and reachable
- [ ] Time-box agreed
- [ ] Communication plan if abort triggers
Post-experiment
Run experiment_postmortem.py --plan <plan.json> --result-log <results> to generate the postmortem.