AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

testing-handbook-generator

Generates Claude Code skills from the Trail of Bits Testing Handbook (appsec.guide), analyzing handbook pages and emitting SKILL.md files with the structure each skill type requires. Use when creating or refreshing a skill from handbook content, or when the user names the testing handbook or appsec.guide. Not for answering security testing questions — the generated skills cover those.

Original source
trailofbits
License
CC-BY-SA-4.0
Pinned revision
2be6433b39e009cd2b6fd3be1895f6b1faa5af9a

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read trailofbits--testing-handbook-generator at revision 2be6433b39e009cd2b6fd3be1895f6b1faa5af9a, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 8

testing.md

Testing Strategy

Methodology for validating generated skills.

Prerequisites

Required tools for validation:

Python Dependencies

None to install: the scripts carry PEP 723 inline metadata, so uv run scripts/<name>.py resolves everything on first use.

Optional Tools

  • yq - YAML processor for manual checks: brew install yq (macOS)

Automated Validation

Use the validator script for comprehensive validation:

# Validate all generated skills
uv run scripts/validate-skills.py

# Validate specific skill
uv run scripts/validate-skills.py --skill libfuzzer

# Output JSON for CI integration
uv run scripts/validate-skills.py --json

# Verbose output with details
uv run scripts/validate-skills.py -v

The validator checks:

  • YAML frontmatter parsing and field validation (name, description)
  • Required sections presence by skill type
  • Line count limits (<500)
  • Hugo shortcode detection
  • Cross-reference validation (related skills exist)
  • Internal link resolution

Path Convention

All paths in this document are relative to testing-handbook-generator/:

  • Generated skills: ../[skill-name]/SKILL.md
  • Templates: ./templates/[type]-skill.md

Quick Validation Reference

Recommended: Use the automated validator:

uv run scripts/validate-skills.py --skill [skill-name]

Manual checks (for debugging or when validator is unavailable):

| Check | Command | Pass | Fail | |-------|---------|------|------| | Line count | wc -l SKILL.md | < 500 | ≥ 500 | | Hugo shortcodes | grep -cE '\{\{[<%]' SKILL.md | 0 | > 0 | | Escaped backticks | grep -cE '\\{3}' SKILL.md| 0 | > 0 | | YAML valid |head -50 SKILL.md | yq -e '.' 2>&1| No output | Error message | | Name format |yq '.name' SKILL.md|lowercase-name| Mixed case, spaces, or special chars | | Description exists |yq '.description' SKILL.md| Non-empty string |nullor empty | | Required sections |grep -c '^## ' SKILL.md` | ≥ 3 | < 3 |

One-Liner Validation

Recommended: Use the automated validator for all checks:

# Validate all skills at once
uv run scripts/validate-skills.py

# JSON output for CI pipelines
uv run scripts/validate-skills.py --json

Legacy shell commands (for environments without Python):

SKILL="../libfuzzer/SKILL.md" && \
  [ $(wc -l < "$SKILL") -lt 500 ] && \
  [ $(grep -cE '\{\{[<%]' "$SKILL") -eq 0 ] && \
  awk '/^---$/{if(++n==2)exit}n==1' "$SKILL" | yq -e '.' >/dev/null 2>&1 && \
  echo "✓ Valid: $SKILL" || echo "✗ Invalid: $SKILL"

Validity Checks

Run these checks on every generated skill before delivery.

1. YAML Frontmatter Validation

Check: Frontmatter parses without errors

# Set skill path (replace 'libfuzzer' with target skill name)
SKILL="../libfuzzer/SKILL.md"

# Extract and validate frontmatter (awk extracts content between first two ---)
awk '/^---$/{if(++n==2)exit}n==1' "$SKILL" | yq -e '.'

# Validate name field
NAME=$(yq '.name' "$SKILL")
echo "$NAME" | grep -qE '^[a-z0-9-]{1,64}$' && echo "Name: OK" || echo "Name: INVALID"

# Validate description length
DESC=$(yq '.description' "$SKILL")
[ ${#DESC} -le 1024 ] && [ ${#DESC} -gt 0 ] && echo "Description: OK" || echo "Description: INVALID"

Required fields: | Field | Requirement | Validation | |-------|-------------|------------| | name | Present, lowercase, max 64 chars | Pattern: ^[a-z0-9-]{1,64}$ | | type | Recommended, one of: tool, fuzzer, technique, domain | Determines required sections | | description | Present, non-empty, max 1024 chars | Length: 1-1024 chars |

Validation rules:

  • No XML/HTML tags in name or description (pattern: <[^>]+>)
  • No reserved words ("anthropic", "claude") in name
  • type field ensures correct section validation (if missing, type is inferred from content)
  • Description leads with what the skill does for the reader, not a definition of the tool, names the concrete commands, flags, and API symbols it covers, and closes with two or more situations in the words a user would type. "Coverage-guided fuzzer built into LLVM. Use for fuzzing C/C++ code" has a what and a when and still routes to nothing — see "Description quality" in agent-prompt.md for the bar.
  • No Hugo shortcodes in frontmatter (pattern: \{\{[<%])

Trigger phrase validation:

The description MUST include a trigger phrase ("Use when" or "Use for"):

# Check for trigger phrase in description
DESC=$(yq '.description' "$SKILL")
echo "$DESC" | grep -qE 'Use when|Use for' && echo "Trigger: OK" || echo "Trigger: MISSING"

2. Required Sections

Check: Skill contains essential sections

| Section | Required | Purpose | |---------|----------|---------| | # Title | Yes | Main heading | | ## When to Use | Yes | Trigger conditions | | ## Quick Reference or ## Quick Start | Yes | Fast access to key info | | Core workflow/content sections | Yes | Main skill content | | ## Resources | If handbook has resources | External links |

Template-specific required sections:

| Skill Type | Required Sections | |------------|-------------------| | Tool | When to Use, Quick Reference, Installation, Core Workflow | | Fuzzer | When to Use, Quick Start, Writing a Harness, Related Skills | | Technique | When to Apply, Quick Reference, Tool-Specific Guidance, Related Skills | | Domain | Background, Quick Reference, Testing Workflow, Related Skills |

Section validation command:

# Validate required sections for a tool skill
SKILL="../semgrep/SKILL.md"
REQUIRED=("When to Use" "Quick Reference" "Installation" "Core Workflow")

for section in "${REQUIRED[@]}"; do
  grep -q "^## $section" "$SKILL" && echo "✓ $section" || echo "✗ Missing: $section"
done

Related Skills validation:

For Fuzzer, Technique, and Domain skills, verify that the Related Skills section exists and references valid skills:

# Check Related Skills section exists
grep -q "^## Related Skills" "$SKILL" || echo "✗ Missing: Related Skills section"

# Extract skill references and check they exist
grep -oE '\*\*[a-z0-9-]+\*\*' "$SKILL" | tr -d '*' | sort -u | while read ref; do
  [ -d "../$ref" ] && echo "✓ $ref exists" || echo "⚠ $ref not found (may be planned)"
done

3. Line Count

Check: SKILL.md under 500 lines

wc -l ../libfuzzer/SKILL.md  # Replace 'libfuzzer' with target skill
# Should be < 500

If over 500 lines:

  • Split into supporting files
  • Keep SKILL.md as overview with decision tree
  • Reference supporting files with relative links

4. Internal Reference Resolution

Check: All internal links resolve

SKILL="../libfuzzer/SKILL.md"  # Replace 'libfuzzer' with target skill
SKILL_DIR=$(dirname "$SKILL")

# Find markdown links and verify they exist
grep -oE '\[.+\]\(([^)]+)\)' "$SKILL" | grep -oE '\([^)]+\)' | tr -d '()' | while read link; do
  # Skip external URLs
  [[ "$link" =~ ^https?:// ]] && continue
  # Check if file exists
  [ -f "$SKILL_DIR/$link" ] && echo "✓ $link" || echo "✗ $link (not found)"
done

Common issues: | Issue | Fix | |-------|-----| | Link to non-existent file | Create file or remove link | | Broken anchor | Update anchor or content | | Absolute path used | Convert to relative path |

5. Code Block Preservation

Check: Code blocks preserved from handbook

Compare code blocks in generated skill vs source handbook section:

  • Indentation preserved
  • Language specifier present
  • No truncation

6. Hugo Shortcode Removal

Check: No Hugo shortcodes remain

# Should return nothing (exit code 1 = pass, exit code 0 = shortcodes found)
grep -E '\{\{[<%]' ../libfuzzer/SKILL.md  # Replace 'libfuzzer' with target skill

Shortcodes to remove:

  • {{< hint >}}...{{< /hint >}}
  • {{< tabs >}}...{{< /tabs >}}
  • {{< customFigure >}}
  • {{% relref "..." %}}

Activation Testing

Verify Claude activates the skill correctly.

Test 1: Direct Invocation

Prompt: "Use the [skill-name] skill"

Expected:

  • Claude loads SKILL.md
  • Claude references skill content
  • Claude follows skill workflow

Test 2: Implicit Trigger

Sample prompts by skill type:

| Skill Type | Test Prompt | |------------|-------------| | Tool (semgrep) | "Scan this Python code for security issues" | | Tool (codeql) | "Set up CodeQL for my Java project" | | Fuzzer (libfuzzer) | "Help me fuzz this C function" | | Fuzzer (aflpp) | "Set up AFL++ for multi-core fuzzing" | | Technique (harness) | "Write a fuzzing harness for this parser" | | Technique (coverage) | "Analyze coverage of my fuzzing campaign" | | Domain (wycheproof) | "Test my ECDSA implementation" | | Domain (constant-time) | "Check if this crypto code has timing leaks" |

Expected:

  • Skill description matches prompt intent
  • Claude selects this skill over alternatives
  • Claude uses skill content appropriately

Test 3: Decision Tree Navigation

Test: Claude navigates to supporting files correctly

Prompts:

  1. General query → Should use SKILL.md overview
  2. Specific topic → Should read appropriate supporting file
  3. Advanced topic → Should follow decision tree to detailed doc

Iteration Loop

┌─────────────────────────────────────────────────────┐
│                    Generate Skill                    │
└─────────────────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────┐
│         Run Validator Script                         │
│  uv run scripts/validate-skills.py --skill [name]   │
│                                                     │
│  Checks: YAML, sections, line count, shortcodes,   │
│          cross-refs, internal links                 │
└─────────────────────────────────────────────────────┘
                         │
              ┌──────────┴──────────┐
              │                     │
         All Pass              Failed
              │                     │
              ▼                     ▼
┌─────────────────────┐   ┌─────────────────────┐
│  Activation Test    │   │   Fix Issues        │
│  - Direct invoke    │   │   - Edit skill      │
│  - Implicit trigger │   │   - Re-run validator│
│  - Decision tree    │   └─────────────────────┘
└─────────────────────┘            │
              │                    │
              ▼                    │
┌─────────────────────┐            │
│     Pass?           │            │
└─────────────────────┘            │
         │                         │
    ┌────┴────┐                    │
   Yes        No                   │
    │         │                    │
    │         └────────────────────┘
    ▼
┌─────────────────────────────────────────────────────┐
│                  Mark Complete                       │
│  - Report success to user                           │
│  - List any warnings from validator                 │
└─────────────────────────────────────────────────────┘

Quality Checklist

Before delivering each generated skill:

## Skill: [name]

### Automated Validation
- [ ] Validator passes: `uv run scripts/validate-skills.py --skill [name]`
- [ ] No errors in output
- [ ] Warnings reviewed and addressed (or documented as acceptable)

### Content Quality (manual review)
- [ ] Description leads with the task, names concrete anchors, and gives two or more situations
- [ ] Every command, flag, file name, and API symbol the description advertises appears in the skill body
- [ ] When to Use section has clear triggers
- [ ] Quick Reference is actionable
- [ ] Code examples are complete and runnable
- [ ] Resources section has titles and URLs

### Activation Testing
- [ ] Direct invocation works
- [ ] Implicit trigger matches expected prompts
- [ ] Decision tree navigation correct

Bulk Generation Report

When generating multiple skills, use the validator JSON output:

# Generate JSON report
uv run scripts/validate-skills.py --json > validation-report.json

# Or view in terminal
uv run scripts/validate-skills.py

Example validator output:

==================================================
VALIDATION REPORT
==================================================
✓ libfuzzer
✓ semgrep ⚠
  WARNING: Line count 480 approaching limit of 500
✗ wycheproof
  ERROR: Missing required sections for domain skill: ['Background']

--------------------------------------------------
Total:    3
Passed:   2
Failed:   1
Warnings: 1
--------------------------------------------------
✗ 1 skill(s) failed validation

After validation, document results:

# Skill Generation Report

## Validator Results
- Total: X
- Passed: Y
- Failed: Z
- With warnings: W

## Actions Needed

### Failed Skills
1. **wycheproof**: Add Background section

### Warnings
1. **semgrep**: Line count 480 - consider splitting

## Next Steps
- [ ] Fix failed skills
- [ ] Address warnings
- [ ] Run activation tests

Common Issues and Fixes

| Issue | Cause | Fix | |-------|-------|-----| | YAML parse error | Unquoted description containing : or # | Put the description on one double-quoted line | | Missing section | Template not fully populated | Fill from handbook | | Over 500 lines | Too much detail in main file | Split to supporting files | | Broken reference | Supporting file not created | Create file or remove link | | Shortcode remains | Incomplete conversion | Apply shortcode stripping | | Activation fails | Poor description | Improve trigger keywords |

Post-Generation Checklist

After all skills pass validation, complete the tasks in SKILL.md Post-Generation Tasks.

Summary of required actions:

  1. Update main README.md with generated skills table
  2. Capture self-improvement notes

Self-Improvement Framework

After each generation run, systematically review and improve the generator.

Review Questions

| Category | Question | If Yes, Update | |----------|----------|----------------| | Content extraction | Did any handbook content not extract cleanly? | discovery.md (section 3.2) | | Shortcodes | Were there shortcodes or formats not handled? | discovery.md (section 3.2) | | Manual fixes | Did any skills require manual fixes after generation? | Templates or agent prompt | | Detection | Are there patterns in the handbook not detected? | discovery.md (section 1.3) | | Activation | Did activation testing reveal description issues? | agent-prompt.md ("Description quality") and the templates | | Validation | Did a bug slip through validation? | testing.md (add new check) |

Improvement Log Format

When making improvements, document them:

## Self-Improvement Log

### YYYY-MM-DD: {Brief title}

**Issue:** {What went wrong during generation}
**Example:** {Specific skill/section affected}
**Root cause:** {Why the current logic failed}
**Fix:** {What was changed}
**Files modified:**
- `{file1}`: {what changed}
- `{file2}`: {what changed}
**Verification:** {How to confirm the fix works}

Improvement Priority

| Priority | Criteria | Action | |----------|----------|--------| | P0 - Critical | Generated skill is broken/unusable | Fix immediately, re-generate affected skills | | P1 - High | Content missing or incorrect | Fix before next generation run | | P2 - Medium | Suboptimal but functional | Add to backlog, fix when convenient | | P3 - Low | Minor formatting/style issues | Optional improvement |