AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

mutation-testing

Configures mewt or muton campaigns, analyzes surviving mutants, and investigates bugs exposed by testing gaps. Use when setting up mutation testing, reviewing campaign results, identifying equivalent mutants, or finding bugs from surviving mutations.

Original source
trailofbits
License
CC-BY-SA-4.0
Pinned revision
2be6433b39e009cd2b6fd3be1895f6b1faa5af9a

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read trailofbits--mutation-testing at revision 2be6433b39e009cd2b6fd3be1895f6b1faa5af9a, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 10

references/optimization-strategies.md

Optimization Strategies

Apply these strategies before running a campaign when Phase 3 of the configuration workflow requires optimization (estimated >16 hours or user requests).


Priority 1: Verify Target Selection

Most common issue: Mutating non-source code.

Diagnostic:

mewt print config     # Check [targets] include/ignore
mewt print targets    # Check what was actually mutated

Look for unintended files:

  • Mocks: src/mocks/, __mocks__/
  • Tests: *_test.rs, *.test.js, tests/
  • Dependencies: vendor/, node_modules/
  • Generated: proto/, generated/

Fix: Update [targets] in mewt.toml to be more specific:

# Before (too broad)
[targets]
include = ["**/*.rs"]

# After (specific)
[targets]
include = ["src/**/*.rs", "lib/**/*.rs"]
ignore = ["test", "mock", "generated"]

Re-run mewt mutate and check new count.


Priority 2: Analyze Project Structure

Goal: Understand mutant distribution and test organization to choose the right optimization.

1. Get mutant counts per component:

# Quote globs to prevent shell expansion. IDs output contains one mutant per line.
mewt print mutants --target 'src/auth/**/*.rs' --format ids | wc -l
mewt print mutants --target 'src/core/**/*.rs' --format ids | wc -l
mewt print mutants --target 'src/utils/**/*.rs' --format ids | wc -l

Present breakdown to user:

Component breakdown:
- src/auth/: 200 mutants × 5s = ~17 min
- src/core/: 800 mutants × 8s = ~1.8 hrs
- src/utils/: 150 mutants × 3s = ~8 min
Total: 1150 mutants, ~2.3 hrs worst-case

2. Count mutations by severity:

# Check enabled mutation types
mewt print config | grep mutations

# Count by severity level
mewt print mutants --severity high --format ids | wc -l
mewt print mutants --severity medium --format ids | wc -l
mewt print mutants --severity low --format ids | wc -l

# Or count specific mutation types
mewt print mutants --mutation-types ER --format ids | wc -l
mewt print mutants --mutation-types CR --format ids | wc -l

# Compare to total
mewt print mutants | wc -l

Example output:

High/Medium severity: 450 mutants
Total mutants: 1200
Percentage: 37.5%

Note: The percentage varies drastically between codebases (15% to 50+ % is common).


Priority 3: Choose Optimization Approach

Based on project structure analysis, present options to user with concrete time estimates:

Option A: Run Full Campaign

  • "Estimated ~X hours worst-case (likely faster in practice)"
  • "Recommend starting Friday evening for weekend completion"
  • When to suggest: Duration acceptable, comprehensive coverage desired

Option B: Target Critical Components

  • "Focus on specific components: src/auth/ (~17 min), src/crypto/ (~45 min)"
  • "Start with one component and expand scope after review?"
  • When to suggest: Clear component boundaries, user wants rapid iteration

Implementation:

[targets]
# Start with critical component
include = ["src/auth/**/*.rs"]

# After review, expand scope
# include = ["src/auth/**/*.rs", "src/core/**/*.rs"]

After editing mewt.toml, purge removed targets then mutate any newly included files: Preserve any results that must be retained and confirm that discarding the affected campaign data is authorized before purging.

mewt purge        # removes targets no longer matching [targets].include/ignore
mewt mutate src/  # adds mutants for any newly included files
mewt status       # confirm reduced mutant count

Option C: High/Medium Severity Only

  • "Limit to high/medium severity mutations (X mutants, ~Y hours)"
  • "Low severity (operator shuffles) tests edge cases, less critical"
  • When to suggest: Time-constrained, need actionable findings quickly

Implementation (by severity level):

[run]
mutations = ["ER", "CR", "IF", "IT"]  # Specific types (high/medium)

After editing mewt.toml, full regeneration is required since existing mutants may no longer be valid under the new filter:

mewt purge --all  # clear all existing mutants
mewt mutate src/  # regenerate with restricted mutation types
mewt status       # confirm reduced mutant count

Or use severity filtering during analysis instead (no database changes needed):

# Run all mutants but filter results by severity
mewt results --severity high,medium
mewt print mutants --severity high

Trade-offs to explain:

  • High/med severity: ~30-40% of mutants (varies by codebase)
  • Low severity: ~60-70% of mutants (operator shuffles, edge cases)
  • Low severity still provides value, just lower priority
  • Using severity filters during analysis allows flexibility without re-running campaign

Option D: Two-Phase Campaign (Integration-Heavy Only)

  • "Phase 1: Targeted tests (estimable upfront), Phase 2: Re-test uncaught with full suite (duration depends on Phase 1 survivor count)"
  • "Total: Phase 1 estimate + (survivors × full-suite time) vs naive total"
  • When to suggest: Integration tests dominate, unit tests don't map cleanly to files

See Two-Phase Campaigns section below for detailed setup.


Two-Phase Campaigns

Use ONLY for integration-heavy test suites. Not recommended for well-organized unit tests.

When to Use

Good fit:

  • Integration tests dominate runtime
  • Unit tests provide broad coverage but don't map cleanly to specific files
  • Targeted test commands significantly faster than full suite

Not recommended:

  • Well-organized unit tests with clear file mappings
  • Tests already fast and targeted

Setup

Phase 1 config (targeted tests):

# TWO-PHASE CAMPAIGN
# Phase 1: Targeted tests (duration estimable upfront)
# Phase 2: Re-test uncaught mutants (duration depends on Phase 1 survivor count)

[test]
# PHASE 2: Uncomment after phase 1 completes
# cmd = "cargo test"
# timeout = 60

# PHASE 1: Targeted tests
[[per_target]]
glob = "src/auth/*.rs"
test.cmd = "cargo test auth::unit"
test.timeout = 10

[[per_target]]
glob = "src/core/*.rs"
test.cmd = "cargo test core::unit"
test.timeout = 15

# Catch-all: full suite for any file not matched above.
# Required unless [targets] is scoped to exactly the globs listed above.
[[per_target]]
glob = "**/*.rs"
test.cmd = "cargo test"
test.timeout = 60

Rationale: Phase 1 uses fast targeted tests. Phase 2 re-tests only the survivors with the comprehensive suite.

Execution

Phase 1:

mewt run

Wait for completion.

Phase 2 (after phase 1 completes):

  1. Extract uncaught mutants:

    mewt results --status Uncaught --format ids > uncaught_ids.txt
    
  2. Update mewt.toml:

    • Comment out all [[per_target]] sections (including the catch-all)
    • Uncomment Phase 2 [test] section
  3. Re-test with full suite:

    mewt test --ids-file uncaught_ids.txt
    
  4. Review final results:

    mewt results  # Remaining uncaught are true coverage gaps
    

Example speedup:

Naive approach:
  2,000 mutants × 45s = 25 hours

Two-phase approach:
  Phase 1: 2,000 mutants × 8s = 4.4 hours → 450 uncaught (example outcome)
  Phase 2: 450 uncaught × 45s = 5.6 hours → 180 truly uncaught
  Total: ~10 hours (2.5× speedup)

Note: Phase 2 duration is unknowable before Phase 1 completes — it depends entirely
on how many mutants survive. The figures above illustrate one possible outcome.
Present Phase 1 as a firm estimate; present Phase 2 as (survivors × full-suite time)
once Phase 1 results are available.

Per-Target Test Configuration

Use when: Tests are well-organized by module/file, and running targeted tests is significantly faster than the full suite.

Setup Pattern

# Test full suite for every mutant (slow but comprehensive)
[test]
cmd = "go test ./..."
timeout = 45

# ALTERNATIVE: Targeted tests per file (fast, may miss cross-module failures)
[[per_target]]
glob = "auth/*.go"
test.cmd = "go test ./auth"
test.timeout = 10

[[per_target]]
glob = "core/*.go"
test.cmd = "go test ./core"
test.timeout = 15

[[per_target]]
glob = "utils/*.go"
test.cmd = "go test ./utils"
test.timeout = 8

# Catch-all for unmatched files
[[per_target]]
glob = "*.go"
test.cmd = "go test ./..."
test.timeout = 45

Ordering matters: First match wins. Place most specific patterns first, catch-all last.

Verify Speedup

time go test ./...      # Full suite: 45s
time go test ./auth     # Targeted: 8s

If targeted tests aren't significantly faster, this optimization won't help.

Trade-offs

Benefits:

  • Faster campaign execution
  • Scales linearly with codebase size

Risks:

  • May miss cross-module integration bugs
  • Requires correct glob-to-test mapping

Mitigation:

  • Use this for initial passes
  • Consider two-phase approach for comprehensive validation