AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

incident-commander

Use when declaring or running a production incident or outage, or writing its post-incident review. Classifies severity with incident_classifier.py, rebuilds the timeline with timeline_reconstructor.py and drafts the PIR with pir_generator.py. Security incidents: incident-response.

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--incident-commander at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 14

references/rca_frameworks_guide.md

Root Cause Analysis (RCA) Frameworks Guide

Overview

This guide provides detailed instructions for applying various Root Cause Analysis frameworks during Post-Incident Reviews. Each framework offers a different perspective and approach to identifying underlying causes of incidents.

Framework Selection Guidelines

| Incident Type | Recommended Framework | Why | |---------------|----------------------|-----| | Process Failure | 5 Whys | Simple, direct cause-effect chain | | Complex System Failure | Fishbone + Timeline | Multiple contributing factors | | Human Error | Fishbone | Systematic analysis of contributing factors | | Extended Incidents | Timeline Analysis | Understanding decision points | | High-Risk Incidents | Bow Tie | Comprehensive barrier analysis | | Recurring Issues | 5 Whys + Fishbone | Deep dive into systemic issues |


5 Whys Analysis Framework

Purpose

Iteratively drill down through cause-effect relationships to identify root causes.

When to Use

  • Simple, linear cause-effect chains
  • Time-pressured analysis
  • Process-related failures
  • Individual component failures

Process Steps

Step 1: Problem Statement

Write a clear, specific problem statement.

Good Example:

"The payment API returned 500 errors for 2 hours on March 15, affecting 80% of checkout attempts."

Poor Example:

"The system was broken."

Step 2: First Why

Ask why the problem occurred. Focus on immediate, observable causes.

Example:

  • Why 1: Why did the payment API return 500 errors?
  • Answer: The database connection pool was exhausted.

Step 3: Subsequent Whys

For each answer, ask "why" again. Continue until you reach a root cause.

Example Chain:

  • Why 2: Why was the database connection pool exhausted?

  • Answer: The application was creating more connections than usual.

  • Why 3: Why was the application creating more connections?

  • Answer: A new feature wasn't properly closing connections.

  • Why 4: Why wasn't the feature properly closing connections?

  • Answer: Code review missed the connection leak pattern.

  • Why 5: Why did code review miss this pattern?

  • Answer: We don't have automated checks for connection pooling best practices.

Step 4: Validation

Verify that addressing the root cause would prevent the original problem.

Best Practices

  1. Ask at least 3 "whys" - Surface causes are rarely root causes
  2. Focus on process failures, not people - Avoid blame, focus on system improvements
  3. Use evidence - Support each answer with data or observations
  4. Consider multiple paths - Some problems have multiple root causes
  5. Test the logic - Work backwards from root cause to problem

Common Pitfalls

  • Stopping too early - First few whys often reveal symptoms, not causes
  • Single-cause assumption - Complex systems often have multiple contributing factors
  • Blame focus - Focusing on individual mistakes rather than system failures
  • Vague answers - Use specific, actionable answers

5 Whys Template

## 5 Whys Analysis

**Problem Statement:** [Clear description of the incident]

**Why 1:** [First why question]
**Answer:** [Specific, evidence-based answer]
**Evidence:** [Supporting data, logs, observations]

**Why 2:** [Second why question]
**Answer:** [Specific answer based on Why 1]
**Evidence:** [Supporting evidence]

[Continue for 3-7 iterations]

**Root Cause(s) Identified:**
1. [Primary root cause]
2. [Secondary root cause if applicable]

**Validation:** [Confirm that addressing root causes would prevent recurrence]

Fishbone (Ishikawa) Diagram Framework

Purpose

Systematically analyze potential causes across multiple categories to identify contributing factors.

When to Use

  • Complex incidents with multiple potential causes
  • When human factors are suspected
  • Systemic or organizational issues
  • When 5 Whys doesn't reveal clear root causes

Categories

People (Human Factors)

  • Training and Skills

    • Insufficient training on new systems
    • Lack of domain expertise
    • Skill gaps in team
    • Knowledge not shared across team
  • Communication

    • Poor communication between teams
    • Unclear responsibilities
    • Information not reaching right people
    • Language/cultural barriers
  • Decision Making

    • Decisions made under pressure
    • Insufficient information for decisions
    • Risk assessment inadequate
    • Approval processes bypassed

Process (Procedures and Workflows)

  • Documentation

    • Outdated procedures
    • Missing runbooks
    • Unclear instructions
    • Process not documented
  • Change Management

    • Inadequate change review
    • Rushed deployments
    • Insufficient testing
    • Rollback procedures unclear
  • Review and Approval

    • Code review gaps
    • Architecture review skipped
    • Security review insufficient
    • Performance review missing

Technology (Systems and Tools)

  • Architecture

    • Single points of failure
    • Insufficient redundancy
    • Scalability limitations
    • Tight coupling between systems
  • Monitoring and Alerting

    • Missing monitoring
    • Alert fatigue
    • Inadequate thresholds
    • Poor alert routing
  • Tools and Automation

    • Manual processes prone to error
    • Tool limitations
    • Automation gaps
    • Integration issues

Environment (External Factors)

  • Infrastructure

    • Hardware failures
    • Network issues
    • Capacity limitations
    • Geographic dependencies
  • Dependencies

    • Third-party service failures
    • External API changes
    • Vendor issues
    • Supply chain problems
  • External Pressure

    • Time pressure from business
    • Resource constraints
    • Regulatory changes
    • Market conditions

Process Steps

Step 1: Define the Problem

Place the incident at the "head" of the fishbone diagram.

Step 2: Brainstorm Causes

For each category, brainstorm potential contributing factors.

Step 3: Drill Down

For each factor, ask what caused that factor (sub-causes).

Step 4: Identify Primary Causes

Mark the most likely contributing factors based on evidence.

Step 5: Validate

Gather evidence to support or refute each suspected cause.

Fishbone Template

## Fishbone Analysis

**Problem:** [Incident description]

### People
**Training/Skills:**
- [Factor 1]: [Evidence/likelihood]
- [Factor 2]: [Evidence/likelihood]

**Communication:**
- [Factor 1]: [Evidence/likelihood]

**Decision Making:**
- [Factor 1]: [Evidence/likelihood]

### Process
**Documentation:**
- [Factor 1]: [Evidence/likelihood]

**Change Management:**
- [Factor 1]: [Evidence/likelihood]

**Review/Approval:**
- [Factor 1]: [Evidence/likelihood]

### Technology
**Architecture:**
- [Factor 1]: [Evidence/likelihood]

**Monitoring:**
- [Factor 1]: [Evidence/likelihood]

**Tools:**
- [Factor 1]: [Evidence/likelihood]

### Environment
**Infrastructure:**
- [Factor 1]: [Evidence/likelihood]

**Dependencies:**
- [Factor 1]: [Evidence/likelihood]

**External Factors:**
- [Factor 1]: [Evidence/likelihood]

### Primary Contributing Factors
1. [Factor with highest evidence/impact]
2. [Second most significant factor]
3. [Third most significant factor]

### Root Cause Hypothesis
[Synthesized explanation of how factors combined to cause incident]

Timeline Analysis Framework

Purpose

Analyze the chronological sequence of events to identify decision points, missed opportunities, and process gaps.

When to Use

  • Extended incidents (> 1 hour)
  • Complex multi-phase incidents
  • When response effectiveness is questioned
  • Communication or coordination failures

Analysis Dimensions

Detection Analysis

  • Time to Detection: How long from onset to first alert?
  • Detection Method: How was the incident first identified?
  • Alert Effectiveness: Were the right people notified quickly?
  • False Negatives: What signals were missed?

Response Analysis

  • Time to Response: How long from detection to first response action?
  • Escalation Timing: Were escalations timely and appropriate?
  • Resource Mobilization: How quickly were the right people engaged?
  • Decision Points: What key decisions were made and when?

Communication Analysis

  • Internal Communication: How effective was team coordination?
  • External Communication: Were stakeholders informed appropriately?
  • Communication Gaps: Where did information flow break down?
  • Update Frequency: Were updates provided at appropriate intervals?

Resolution Analysis

  • Mitigation Strategy: Was the chosen approach optimal?
  • Alternative Paths: What other options were considered?
  • Resource Allocation: Were resources used effectively?
  • Verification: How was resolution confirmed?

Process Steps

Step 1: Event Reconstruction

Create comprehensive timeline with all available events.

Step 2: Phase Identification

Identify distinct phases (detection, triage, escalation, mitigation, resolution).

Step 3: Gap Analysis

Identify time gaps and analyze their causes.

Step 4: Decision Point Analysis

Examine key decision points and alternative paths.

Step 5: Effectiveness Assessment

Evaluate the overall effectiveness of the response.

Timeline Template

## Timeline Analysis

### Incident Phases
1. **Detection** ([start] - [end], [duration])
2. **Triage** ([start] - [end], [duration])
3. **Escalation** ([start] - [end], [duration])
4. **Mitigation** ([start] - [end], [duration])
5. **Resolution** ([start] - [end], [duration])

### Key Decision Points
**[Timestamp]:** [Decision made]
- **Context:** [Situation at time of decision]
- **Alternatives:** [Other options considered]
- **Outcome:** [Result of decision]
- **Assessment:** [Was this optimal?]

### Communication Timeline
**[Timestamp]:** [Communication event]
- **Channel:** [Slack/Email/Phone/etc.]
- **Audience:** [Who was informed]
- **Content:** [What was communicated]
- **Effectiveness:** [Assessment]

### Gaps and Delays
**[Time Period]:** [Description of gap]
- **Duration:** [Length of gap]
- **Cause:** [Why did gap occur]
- **Impact:** [Effect on incident response]

### Response Effectiveness
**Strengths:**
- [What went well]
- [Effective decisions/actions]

**Weaknesses:**
- [What could be improved]
- [Missed opportunities]

### Root Causes from Timeline
1. [Process-based root cause]
2. [Communication-based root cause]
3. [Decision-making root cause]

Bow Tie Analysis Framework

Purpose

Analyze both preventive measures (left side) and protective measures (right side) around an incident.

When to Use

  • High-severity incidents (SEV1)
  • Security incidents
  • Safety-critical systems
  • When comprehensive barrier analysis is needed

Components

Hazards

What conditions create the potential for incidents?

Examples:

  • High traffic loads
  • Software deployments
  • Human interactions with critical systems
  • Third-party dependencies

Top Event

What actually went wrong? This is the center of the bow tie.

Examples:

  • "Database became unresponsive"
  • "Payment processing failed"
  • "User authentication service crashed"

Threats (Left Side)

What specific causes could lead to the top event?

Examples:

  • Code defects in new deployment
  • Database connection pool exhaustion
  • Network connectivity issues
  • DDoS attack

Consequences (Right Side)

What are the potential impacts of the top event?

Examples:

  • Revenue loss
  • Customer churn
  • Regulatory violations
  • Brand damage
  • Data loss

Barriers

What controls exist (or could exist) to prevent threats or mitigate consequences?

Preventive Barriers (Left Side):

  • Code reviews
  • Automated testing
  • Load testing
  • Input validation
  • Rate limiting

Protective Barriers (Right Side):

  • Circuit breakers
  • Failover systems
  • Backup procedures
  • Customer communication
  • Rollback capabilities

Process Steps

Step 1: Define the Top Event

Clearly state what went wrong.

Step 2: Identify Threats

Brainstorm all possible causes that could lead to the top event.

Step 3: Identify Consequences

List all potential impacts of the top event.

Step 4: Map Existing Barriers

Identify current controls for each threat and consequence.

Step 5: Assess Barrier Effectiveness

Evaluate how well each barrier worked (or failed).

Step 6: Recommend Additional Barriers

Identify new controls needed to prevent recurrence.

Bow Tie Template

## Bow Tie Analysis

**Top Event:** [What went wrong]

### Threats (Potential Causes)
1. **[Threat 1]**
   - Likelihood: [High/Medium/Low]
   - Current Barriers: [Preventive controls]
   - Barrier Effectiveness: [Assessment]

2. **[Threat 2]**
   - Likelihood: [High/Medium/Low]
   - Current Barriers: [Preventive controls]
   - Barrier Effectiveness: [Assessment]

### Consequences (Potential Impacts)
1. **[Consequence 1]**
   - Severity: [High/Medium/Low]
   - Current Barriers: [Protective controls]
   - Barrier Effectiveness: [Assessment]

2. **[Consequence 2]**
   - Severity: [High/Medium/Low]
   - Current Barriers: [Protective controls]
   - Barrier Effectiveness: [Assessment]

### Barrier Analysis
**Effective Barriers:**
- [Barrier that worked well]
- [Why it was effective]

**Failed Barriers:**
- [Barrier that failed]
- [Why it failed]
- [How to improve]

**Missing Barriers:**
- [Needed preventive control]
- [Needed protective control]

### Recommendations
**Preventive Measures:**
1. [New barrier to prevent threat]
2. [Improvement to existing barrier]

**Protective Measures:**
1. [New barrier to mitigate consequence]
2. [Improvement to existing barrier]

Framework Comparison

| Framework | Time Required | Complexity | Best For | Output | |-----------|---------------|------------|----------|---------| | 5 Whys | 30-60 minutes | Low | Simple, linear causes | Clear cause chain | | Fishbone | 1-2 hours | Medium | Complex, multi-factor | Comprehensive factor map | | Timeline | 2-3 hours | Medium | Extended incidents | Process improvements | | Bow Tie | 2-4 hours | High | High-risk incidents | Barrier strategy |

Combining Frameworks

5 Whys + Fishbone

Use 5 Whys for initial analysis, then Fishbone to explore contributing factors.

Timeline + 5 Whys

Use Timeline to identify key decision points, then 5 Whys on critical failures.

Fishbone + Bow Tie

Use Fishbone to identify causes, then Bow Tie to develop comprehensive prevention strategy.

Quality Checklist

  • [ ] Root causes address systemic issues, not symptoms
  • [ ] Analysis is backed by evidence, not assumptions
  • [ ] Multiple perspectives considered (technical, process, human)
  • [ ] Recommendations are specific and actionable
  • [ ] Analysis focuses on prevention, not blame
  • [ ] Findings are validated against incident timeline
  • [ ] Contributing factors are prioritized by impact
  • [ ] Root causes link clearly to preventive actions

Common Anti-Patterns

  • Human Error as Root Cause - Dig deeper into why human error occurred
  • Single Root Cause - Complex systems usually have multiple contributing factors
  • Technology-Only Focus - Consider process and organizational factors
  • Blame Assignment - Focus on system improvements, not individual fault
  • Generic Recommendations - Provide specific, measurable actions
  • Surface-Level Analysis - Ensure you've reached true root causes

Last Updated: February 2026
Next Review: August 2026
Owner: SRE Team + Engineering Leadership