references/incident-response-framework.md · 24000
rable, fail over to the secondary. Test failover procedures quarterly so this is a routine, not a gamble.
- Customer workaround: If mitigation will take time, publish a workaround for customers (e.g., "Use the mobile app while we restore web access").
Mitigation verification: After applying mitigation, monitor key metrics for 15 minutes before declaring the issue mitigated. Premature declarations that the issue is mitigated followed by recurrence damage team credibility and customer trust.
Phase 6: Resolution
Resolution is declared when the root cause is addressed and service is operating normally:
- Verification checklist: Error rates returned to baseline, latency returned to baseline, no ongoing customer reports, monitoring confirms stability for 30+ minutes.
- Incident channel update: IC posts final status with resolution summary, total duration, and next steps.
- Status page update: Post resolution notice within 15 minutes of declaring resolved.
- Stand down: IC explicitly releases all responders. SMEs return to normal work. Bridge call is closed.
Phase 7: Postmortem
Postmortem is mandatory for SEV1 and SEV2. Optional but recommended for SEV3. Never conducted for SEV4.
- Timeline: Postmortem document drafted within 24 hours. Postmortem meeting held within 72 hours (3 business days). Action items assigned and tracked in the team's issue tracker.
- Blameless standard: The postmortem examines systems, processes, and tools -- not individual performance. "Why did the system allow this?" not "Why did [person] do this?"
- Required sections: Timeline (from scribe's log), root cause analysis (using 5 Whys or fault tree), impact summary (users, revenue, duration), what went well, what went poorly, action items with owners and due dates.
- Action items and recurrence: Every postmortem produces 3-7 concrete action items. Items without owners and due dates are not action items. Teams should close 80%+ within 30 days. If the same root cause appears in two postmortems within 6 months, escalate to engineering leadership as a systemic reliability investment area.