AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

chaos-engineering

Use before running a chaos experiment, gameday or fault injection, or after one to write it up. Designs it with steady state and abort criteria in experiment_designer.py, sizes the blast radius with blast_radius_calculator.py and drafts the postmortem.

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--chaos-engineering at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 7

references/tooling_landscape.md

Tooling landscape

Six options. Pick by stack, license preference, and required attack types.

At-a-glance

| Tool | License | Stack | Attack coverage | Best for | |---|---|---|---|---| | Chaos Toolkit | OSS (Apache 2) | Any (Python) | Broad via plugins | Lightweight, multi-cloud, JSON experiments | | Chaos Mesh | OSS (Apache 2) | Kubernetes | Very broad (network, pod, IO, time, stress) | k8s-native, rich CRDs | | Litmus | OSS (Apache 2) | Kubernetes | Very broad (300+ experiments) | k8s, Argo-integrated | | Gremlin | Commercial | Any (agents) | Broad, polished | Enterprise, audit, multi-cloud | | AWS FIS | Paid (AWS) | AWS | AWS services + EC2/ECS/EKS | AWS-heavy, IAM-integrated | | Custom | Your code | Any | What you build | Niche, single-cloud, low budget |

Decision tree

Stack constraint?
├── Kubernetes-only ──┬── OSS preferred → Chaos Mesh OR Litmus
│                     │       (Litmus has the bigger experiment library;
│                     │        Chaos Mesh has cleaner CRD model)
│                     └── Enterprise budget → Gremlin
│
├── AWS-heavy ────────┬── Simple needs → AWS FIS
│                     ├── Multi-cloud + AWS → Chaos Toolkit + AWS plugin
│                     └── Enterprise → Gremlin
│
├── Multi-cloud ──────┬── OSS → Chaos Toolkit
│                     └── Enterprise → Gremlin
│
└── No infra constraint
    └── Just need fault injection → Toxiproxy (a single-purpose tool, not full chaos framework)

Chaos Toolkit

What it is: Python-based framework. You write experiments as JSON or YAML files; the CLI runs them.

Strengths:

  • Lightweight; runs anywhere Python runs
  • Plugin ecosystem for AWS, Azure, GCP, Kubernetes, etc.
  • JSON experiments are version-controllable
  • Apache 2 license

Weaknesses:

  • No built-in scheduling (you bring cron / CI)
  • Smaller experiment library than Litmus
  • Plugin quality varies

Example experiment (JSON):

{
  "title": "Latency on payment-svc",
  "description": "p99 latency stays <500ms when payment is +200ms slow",
  "steady-state-hypothesis": {
    "title": "p99 < 500ms",
    "probes": [{ "type": "probe", "tolerance": [0, 500],
                 "provider": { "type": "http", "url": "https://my.dashboards/p99" } }]
  },
  "method": [{ "type": "action", "name": "add-latency",
               "provider": { "type": "process", "path": "tc", "arguments": [...] } }]
}

Chaos Mesh

What it is: Kubernetes operator + CRDs for chaos. Install in-cluster; kubectl apply an experiment.

Strengths:

  • True k8s-native (no external orchestrator)
  • Comprehensive coverage: network, pod, IO, stress, time, DNS, HTTP, kernel
  • UI dashboard for running experiments
  • CNCF Incubating project

Weaknesses:

  • k8s-only
  • CRD layout is opinionated; some types feel similar but aren't
  • Setup requires cluster admin

Example experiment (CRD):

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-latency
spec:
  action: delay
  mode: one
  selector:
    namespaces: [default]
    labelSelectors:
      app: payment-svc
  delay:
    latency: 200ms
  duration: 5m

Litmus

What it is: Kubernetes chaos framework with a large experiment library. Argo-CD integration.

Strengths:

  • 300+ pre-built experiments
  • Strong Argo / GitOps integration
  • ChaosHub community library
  • Workflow capability for multi-step experiments

Weaknesses:

  • More moving parts than Chaos Mesh
  • Some pre-built experiments are thin wrappers; quality varies
  • k8s-only

Gremlin

What it is: Commercial SaaS. Agents on hosts; central control plane.

Strengths:

  • Polished UX
  • Comprehensive attack library
  • Audit logs (compliance)
  • Multi-cloud, multi-OS
  • Customer support

Weaknesses:

  • Paid (per-host or per-MAU)
  • Vendor lock-in
  • Less control than OSS

When to choose: large enterprise, compliance/audit requirements, dedicated chaos team, budget exists.

AWS FIS (Fault Injection Simulator)

What it is: AWS-managed chaos service. Templates of "actions" (stop instance, throttle API) chained into experiments.

Strengths:

  • IAM-integrated (proper auth/audit)
  • Native to AWS services (RDS failover, ECS/EKS, Network Manager)
  • Pay-per-experiment (no agents to maintain)

Weaknesses:

  • AWS-only
  • Smaller attack library than Chaos Mesh / Gremlin
  • Multi-account is awkward

When to choose: AWS-heavy team that wants chaos without managing the chaos infra.

Custom (DIY)

When to choose:

  • Single-cloud, single-stack, low complexity
  • Budget = $0
  • Have engineering capacity to maintain the tool
  • Need a niche attack type that no tool covers

Implementation patterns:

  • Bash scripts that wrap tc / iptables / kill / stress-ng
  • Application-level chaos via feature flags + middleware
  • Service mesh fault injection (Istio / Linkerd) — covers many cases without a chaos framework

Trade-offs:

  • You build all the safety rails (abort, timeout, blast-radius)
  • You build the scheduler
  • You debug your own bugs

For most teams, this is a starter path; once chaos becomes regular, switch to a real tool.

Pricing rule of thumb

| Tool | Typical cost (annual) | |---|---| | Chaos Toolkit | $0 | | Chaos Mesh | $0 | | Litmus OSS | $0 | | Litmus Enterprise | $5-30k | | Gremlin | $20-100k+ | | AWS FIS | pay-per-action, ~$100-2000/mo for active use | | Custom | engineering time only |

Migration paths

| From | To | Effort | |---|---|---| | Custom scripts | Chaos Toolkit | Low (wrap scripts as actions) | | Chaos Toolkit | Chaos Mesh | Medium (k8s-only; rewrite for CRDs) | | Chaos Mesh | Litmus | Medium (similar shape, different CRDs) | | Anything | Gremlin | Easy (Gremlin imports many formats) |

Selection checklist

Before committing:

  • [ ] Stack matches (k8s vs multi-cloud vs AWS-only)
  • [ ] Required attack types covered (cross-reference attack_taxonomy.md)
  • [ ] Audit logging requirement met (Gremlin / AWS FIS only have full audit)
  • [ ] Self-hosting requirement met (OSS only)
  • [ ] Budget approved
  • [ ] Run a 30-day proof-of-concept; verify abort path works