references/verification-checklist.md · 48000
ng signal of artificial padding.
- [ ] N/A entries excluded from totals: If any component has
N/A — {justification}entries for STRIDE categories, verify those categories show 0 in the Summary table (not 1). N/A entries do NOT count as threats.
3.7 Sort Order (Findings)
- [ ] Within each tier section: Findings appear in order Critical → Important → Moderate → Low
- [ ] Within each severity band: Higher-CVSS findings appear before lower-CVSS findings
- [ ] No misordering: Scan sequentially and confirm no reversal
3.8 Report Files Table (Assessment ↔ Output Folder)
- [ ] Every file listed in the Report Files table of
0-assessment.mdexists in the output folder - [ ]
0.1-architecture.mdis listed in the Report Files table - [ ] If
1.2-threatmodel-summary.mmdwas not generated: it is omitted from the Report Files table (not listed with a "N/A" note)
Phase 4 — Evidence Quality Checks
These checks validate the substance of findings, not just structure. Ideally run by a sub-agent with code access.
4.1 Finding Evidence
- [ ] Every finding has an Evidence section citing specific files/lines/configs
- [ ] Evidence is concrete: Shows actual code or config, not just "absence of config"
- [ ] For "missing security" claims: Evidence proves the platform default is insecure (not just that explicit config is absent)
4.2 Verify-Before-Flagging Compliance
- [ ] Security infrastructure inventory was performed before STRIDE analysis (check for platform security defaults verification in findings)
- [ ] No false positive patterns: No finding claims "missing mTLS" when Dapr Sentry is present, or "missing RBAC" on K8s ≥1.6, etc.
- [ ] Finding classification applied: Every documented finding is "Confirmed" (not "Needs Verification" — those belong in
0-assessment.md)
4.3 Needs Verification Placement
- [ ] All "Needs Verification" items are in
0-assessment.mdunder Analysis Context & Assumptions — NOT in3-findings.md - [ ] No ambiguous findings: Findings in
3-findings.mdhave positive evidence of a vulnerability
Verification Summary Template
After running all checks, produce a summary.
Sub-agent output MUST include:
- Phase name
- Total checks, Passed, Failed
- For each failure: Check ID, file, evidence, exact fix instruction
- Re-run status after fixes
Do not return "looks good" without counts.
## Verification Results
| Phase | Checks | Passed | Failed | Notes |
|-------|--------|--------|--------|-------|
| 0 — Common Deviation Scan | [N] | [N] | [N] | [pattern matches] |
| 1 — Per-File Structural | [N] | [N] | [N] | [files with issues] |
| 2 — Diagram Rendering | [N] | [N] | [N] | [specific failures] |
| 3 — Cross-File Consistency | [N] | [N] | [N] | [gaps found] |
| 4 — Evidence Quality | [N] | [N] | [N] | [false positive risks] |
| 5 — JSON Schema | [N] | [N] | [N] | [schema issues] |
### Failed Checks Detail
<!-- For each failed check, list: check ID, file(s), what's wrong, suggested fix -->
Phase 5 — threat-inventory.json Schema Validation
These checks validate the JSON inventory file generated in Step 8b. This file is critical for comparison mode.
5.1 Schema Fields
- [ ]
schema_versionfield — Present and equals"1.0"(standalone) or"1.1"(incremental). If the report contains"incremental": true, schema_version MUST be"1.1". Otherwise"1.0". - [ ]
commitfield — Present (short SHA or"Unknown") - [ ]
componentsarray — Non-empty, has at least 1 entry - [ ] Component IDs — Every component has
id(PascalCase),display,type,boundary - [ ] Component field name compliance — Components use
"display"(NOT"display_name"). Grep:"display_name"must return 0 matches. - [ ] Threat field name compliance — Threats use
"stride_category"(NOT"category"). Threats have BOTH"title"AND"description"(NOT justdescriptionalone, NOT"name"). Threat→component link is inside"identity_key"."component_id"(NOT a top-level"component_id"on the threat object). Grep: top-level"category":outside identity_key must return 0 matches. Grep: every threat object must contain"title":. - [ ]
boundariesarray — Present (can be empty for flat systems) - [ ]
flowsarray — Present, each flow has canonical ID formatDF_{Source}_to_{Target} - [ ]
threatsarray — Non-empty - [ ]
findingsarray — Non-empty - [ ]
metricsobject — Present withtotal_components,total_threats,total_findings
5.2 Metrics Consistency
- [ ]
metrics.total_components == components.length— Array length matches count - [ ]
metrics.total_threats == threats.length— Array length matches count - [ ]
metrics.total_findings == findings.length— Array length matches count - [ ] Metrics match markdown reports —
total_threatsequals Total from STRIDE summary table,total_findingsequals### FIND-count in3-findings.md - [ ] Truncation recovery gate — If ANY array length mismatch was detected above, verify that the file was regenerated (not patched). Check: file size > 10KB for repos with >40 threats; threats array has entries for EVERY component that appears in
2-stride-analysis.md - [ ] Pre-write strategy compliance — If
metrics.total_threats > 50, verify that the JSON was written via sub-agent delegation, Python script, or chunked append — NOT a singlecreate_filecall. Evidence: check log foragentinvocation or_extract.pyscript or multiplereplace_string_in_fileoperations on the JSON file.
5.3 Deterministic Identity Stability (for comparison readiness)
- [ ] Components include deterministic identity fields — every component has
aliases(array),boundary_kind, andfingerprint - [ ]
boundary_kindvalid values — every component'sboundary_kindis one of:MachineBoundary,NetworkBoundary,ClusterBoundary,ProcessBoundary,PrivilegeBoundary,SandboxBoundary. ❌ Any other value (e.g.,DataStorage,ApplicationCore,deployment,trust) → FAIL - [ ] Boundaries include deterministic identity fields — every boundary has
kind,aliases(array), andcontains_fingerprint - [ ] Boundary
kindvalid values — every boundary'skindis one of the same 6 TMT-aligned values asboundary_kind. ❌ Any other value → FAIL - [ ] No duplicate canonical component IDs —
components[].idvalues are unique after normalization - [ ] Alias mapping is coherent — no alias appears under two unrelated component IDs in the same inventory
- [ ] Fingerprint evidence fields are stable-only —
fingerprintuses source files/topology/type/protocols, not freeform prose - [ ] Deterministic ordering applied — arrays sorted by canonical key (
components.id,boundaries.id,flows.id,threats.id,findings.id)
5.4 Comparison Drift Guardrails (when validating comparison outputs)
- [ ] High-confidence rename candidates are not left as add/remove — component pairs with strong alias/source-file/topology overlap are classified as
renamed/modified - [ ] Boundary rename candidates use containment overlap — same
kind+ highcontainsoverlap are classified as boundaryrenamed, notadded+removed - [ ] Split/merge boundary transitions recognized — one-to-many and many-to-one containment transitions are mapped to
split/mergedcategories
5.5 Comparison Integrity Checks (when validating comparison outputs)
- [ ] Baseline ≠ Current commit —
metadata.json→baseline.commitmust differ fromcurrent.commit. Same-commit comparisons are invalid (zero real code changes to compare). - [ ] Files changed > 0 —
metadata.json→git_diff_stats.files_changedmust be > 0. A comparison with 0 files changed has no code delta and is meaningless. - [ ] Duration > 0 —
metadata.json→durationmust NOT be"0m 0s"or any value under 2 minutes. A genuine comparison requires reading two inventories, performing multi-signal matching, computing heatmaps, and generating HTML — this takes real time. - [ ] No external folder references —
metadata.jsonand all output files must NOT contain references toD:\One\tmor any folder outside the repository being analyzed. Reports should only reference folders within the current repo. - [ ] Anti-reuse verification — The comparison output must be freshly generated, not copied from a prior
threat-model-compare-*folder. Verify by checking thatmetadata.jsontimestamps are from the current run. - [ ] Methodology drift ratio — If
diff-result.json→metrics.methodology_drift_ratio> 0.50, verify the HTML report contains a methodology drift warning banner. If ratio not computed but >50% of component renames share the same aliases/fingerprints, flag as validation failure.
Phase 6 — Deterministic Identity & Naming Stability
These checks validate that component/boundary/flow naming follows deterministic rules, enabling reproducible outputs across independent runs of the same code.
6.1 Component ID Determinism
- [ ] Component IDs derived from code artifacts — Every component ID in
threat-inventory.jsonmust trace to an actual class name, file path, deployment manifestmetadata.name, or config key. No abstract concepts (ConfigurationStore,DataLayer,LocalFileSystem). Grep component IDs against source file names and class names — at least 80% should have a direct match. - [ ] Component anchor verification — Every process-type component in
threat-inventory.jsonmust have non-emptyfingerprint.source_filesorfingerprint.source_directories. If both are empty → FAIL (component has no code anchor). - [ ] Helm/K8s workload naming — For K8s-deployed components, verify the component ID matches the
metadata.namefrom the Deployment/StatefulSet YAML, not the Helm template filename or directory. Example:DevPortal(from deployment name), NOTtemplates-knowledge-deployment(from file path). - [ ] External service anchoring — External services (no source code in repo) must anchor to their integration point: client class name, config key, or SDK dependency. Verify
fingerprint.config_keysorfingerprint.class_namesis populated. - [ ] Forbidden naming patterns absent — No component ID is a generic label: grep for
ConfigurationStore,DataLayer,LocalFileSystem,SecurityModule,NetworkLayer,DatabaseAccess. → Must return 0 matches. - [ ] Acronym consistency — Well-known acronyms must be ALL-CAPS in PascalCase IDs:
API,NFS,LLM,SQL,DB,AD,UI. Grep forApi(should beAPI),Nfs(should beNFS),Llm(should beLLM). → Must return 0 matches. - [ ] Common technology naming exactness — Verify these exact IDs where applicable:
Redis(notRedisCache),Milvus(notMilvusDB),NginxIngress(notIngressNginx),AzureAD(notAzureAd),PostgreSQL(notPostgres).
6.2 Boundary Naming Stability
- [ ] Boundary IDs are PascalCase — Every boundary ID in
threat-inventory.jsonuses PascalCase derived from deployment topology (e.g.,K8sCluster,External,Application). NOT code architecture layers (PresentationLayer,BusinessLogic). - [ ] No code-layer boundaries for single-process apps — If the system is a single process (one .exe, one container), there should be exactly 1
Applicationboundary — NOT 4+ boundaries for Presentation/Business/Data layers. Count boundaries and verify proportion. - [ ] K8s multi-service sub-boundaries — For K8s namespaces with multiple Deployments, verify sub-boundaries exist:
BackendServices,DataStorage,MLModels,Agentic(as applicable).
6.3 Data Flow Completeness
- [ ] Bidirectional flows for ingress/reverse proxy — If an ingress component (Nginx, Traefik) routes to backends, verify BOTH directions exist:
DF_Ingress_to_BackendANDDF_Backend_to_Ingress. Count forward flows through ingress and verify matching response flows. - [ ] Bidirectional flows for databases — For every
DF_Service_to_Datastoreflow, verify a correspondingDF_Datastore_to_Serviceread flow exists. Datastores: Redis, Milvus, PostgreSQL, MongoDB, etc. - [ ] Flow count stability — Count flows in
threat-inventory.json. Two independent runs on same code should produce same count (±3 acceptable). If flow count differs by >5 between old and HEAD analyses for unchanged components, flag as naming drift.
6.4 Count Stability (Cross-Run Determinism)
- [ ] Component count within tolerance — If comparing two analyses of the same code, component count must be within ±1. Difference ≥3 = FAIL.
- [ ] Boundary count within tolerance — Same code → boundary count within ±1.
- [ ] Fingerprint completeness for process components — Every component with
type: "process"must have non-emptyfingerprint.source_directoriesandfingerprint.class_names. Empty arrays for process components → FAIL. - [ ] STRIDE category single-letter enforcement — Every
threats[].stride_categoryin JSON is exactly one letter: S, T, R, I, D, E, or A. Grep for full names ("Spoofing","Tampering","Denial of Service") → Must return 0 matches. This prevents heatmap computation errors.
Phase 7 — Evidence-Based Prerequisites & Coverage Completeness
These checks validate that prerequisites, tiers, and coverage follow deterministic evidence-based rules.
7.1 Prerequisite Determination Evidence
- [ ] No prerequisite without deployment evidence — For every finding with
Exploitation Prerequisites≠None, verify the prerequisite reflects actual deployment config (Helm values, Dockerfile, service type, ingress rules). If prerequisite saysInternal Networkbut no evidence of network restriction exists → FAIL. - [ ] Prerequisite consistency across same code — If two analyses of the same code produce different prerequisites for the same vulnerability, the skill rules are insufficient. Flag for investigation.
7.1b Deployment Classification Gate (MANDATORY)
- [ ] Deployment Classification present —
0.1-architecture.mdmust contain a**Deployment Classification:**line with one of:LOCALHOST_DESKTOP,LOCALHOST_SERVICE,AIRGAPPED,K8S_SERVICE,NETWORK_SERVICE. ❌ Missing → FAIL. - [ ] Component Exposure Table present —
0.1-architecture.mdmust contain a### Component Exposure Tablewith columns: Component, Listens On, Auth Required, Reachability, Min Prerequisite, Derived Tier. ❌ Missing → FAIL. - [ ] Exposure table completeness — Every component in Key Components table has a corresponding row in the Component Exposure Table. ❌ Missing rows → FAIL.
- [ ] Deployment classification enforced on T1 — If Deployment Classification is
LOCALHOST_DESKTOPorLOCALHOST_SERVICE:- Count findings with
Exploitation Prerequisites=None. ❌ Count > 0 → FAIL (must beLocal Process AccessorHost/OS Accessminimum). - Count findings in
## Tier 1. ❌ Count > 0 → FAIL (must be T2+ for localhost/desktop apps). - For each finding with
AV:Nin CVSS, check the component'sReachabilitycolumn. ❌AV:NwithReachability ≠ External→ FAIL.
- Count findings with
- [ ] Prerequisite floor enforced — For EVERY finding, look up the finding's
Componentin the exposure table. The finding'sExploitation Prerequisitesmust be ≥ theMin Prerequisitein the table. The finding's tier must be ≥ theDerived Tier. ❌ Finding hasNonebut table saysLocal Process Access→ FAIL. - [ ] Prerequisite basis in Evidence — Every finding's
#### Evidencesection must contain a**Prerequisite basis:**line citing specific code/config that determines the prerequisite. ❌ Missing or generic ("found in codebase") → FAIL.
7.2 Coverage Completeness
- [ ] Technology coverage check — For each major technology in the repo (Redis, PostgreSQL, Docker, K8s, ML/LLM, NFS, etc.), verify at least one finding or documented mitigation addresses it. Scan
0.1-architecture.mdTechnology Stack table → for each technology, grep3-findings.mdfor a matching finding. - [ ] Minimum finding threshold — Small repo (<20 files): ≥8 findings; Medium (20-100): ≥12; Large (100+): ≥18. Count
### FIND-headings and verify against repo size. - [ ] Platform ratio within context-aware limit — Detect deployment pattern: if go.mod contains
controller-runtime/kubebuilder/operator-sdk→ K8s Operator (limit ≤35%); otherwise → Standalone App (limit ≤20%). Count Platform-status threats / total threats. If exceeds limit → FAIL. Document detected pattern in assessment. - [ ] DoS with None prerequisites = Finding — Every DoS threat (
.D) withPrerequisites: Nonemust have a corresponding finding. Grep STRIDE analysis for.Dthreats with None prerequisites and verify each maps to a finding ID in Coverage table.
7.3 Security Infrastructure Awareness
- [ ] Security infrastructure inventory mentioned — Verify
0.1-architecture.mdor2-stride-analysis.mdreferences security components (service mesh, cert management, auth middleware) if they exist in the codebase. If Dapr Sentry is deployed, mTLS cannot be flagged as "missing." - [ ] Burden of proof for missing-security claims — Every finding that claims "missing X" must prove the platform default is insecure, not just that explicit config is absent. Spot-check the highest-severity "missing" finding.
Phase 8 — Comparison HTML Report Structure (comparison outputs only)
These checks validate the HTML comparison report structure.
8.1 HTML Comparison Report Structure
- [ ] Exactly 4
<h2>sections — The HTML must have exactly these<h2>headings in order: "Executive Summary", "Threat Tier Distribution", "STRIDE-A Heatmap (with Delta Indicators)", "Comparison Basis — Component Mapping". ❌ Extra sections like "Overall Risk Shift", "Key Delta Metrics", "Metrics Overview", "Findings Diff" as<h2>→ FAIL (these are either inline elements or removed). ❌ Missing any of the 4 → FAIL. - [ ] No Findings Diff section — The HTML must NOT contain a "Findings Diff"
<h2>section or any findings diff subsections (Fixed, Removed, Analysis Gaps, New, Changed, Unchanged). If present → FAIL. - [ ] No delta metric cards — The HTML must NOT contain
.risk-deltacards (Findings Fixed, New Findings, Net Change, Removed, Analysis Gaps, Code-Verified). If present → FAIL. - [ ] Risk shift and metrics bar as inline elements — Risk shift and metrics bar (Components/Threats/Boundaries/Flows/Time) are inline card elements, NOT
<h2>sections. If they appear as<h2>→ FAIL. - [ ] Metrics bar includes trust boundaries — The metrics bar MUST show trust boundary counts (e.g.,
2 → 2). If boundaries are missing from the metrics bar → FAIL. Components, Threats, Trust Boundaries, Findings, and Code Changes are the 5 required metric boxes. - [ ] Metrics bar 5th box is Code Changes — The 5th metrics box MUST show commit count and PR count (e.g.,
142 commits, 23 PRs). ❌ "Time Between" → FAIL. The duration/dates are now in the comparison cards (Section 1), not the metrics bar. - [ ] Comparison cards structure — Section 1 MUST contain a
comparison-cardsdiv with 3 sub-cards: Baseline (hash, date, rating), Target (hash, date, rating), Trend (direction, duration). ❌ Old-stylesubtitlediv withBaseline: SHA → Target: SHA→ FAIL. ❌ Separaterisk-shiftdiv → FAIL (merged into comparison cards). - [ ] No duplicate status indicators — Status information (Fixed/New/Previously Unidentified counts) MUST appear in ONLY ONE place: the colored status summary cards. They MUST NOT also appear as small inline badges or text in the metrics bar. If the same counts appear in both the metrics bar AND colored cards → FAIL (remove from metrics bar, keep colored cards).
- [ ] Tier labels match analysis reports — The Threat Tier Distribution section in the HTML must use EXACTLY these labels: "Tier 1 — Direct Exposure", "Tier 2 — Conditional Risk", "Tier 3 — Defense-in-Depth". ❌ "Probable Exposure", "Theoretical", "High Risk", or any invented variant → FAIL.
- [ ] Section title is "Comparison Basis" not "Architecture Changes" — The component mapping section must be titled "Comparison Basis — Component Mapping", NOT "Architecture Changes".
- [ ] Heatmap has 13 columns — The STRIDE-A heatmap grid must have: Component | S | T | R | I | D | E | A | Total | divider | T1 | T2 | T3. If T1/T2/T3 columns are missing → FAIL. The heatmap title must include "(with Delta Indicators)".
8.2 Heatmap Accuracy (comparison outputs)
- [ ] Heatmap not all zeros — Sum all
baseline.Totalandcurrent.Totalinstride_heatmap.components. If either sum is 0 but corresponding inventory has threats → FAIL (heatmap computation bug). - [ ] No duplicate renamed component rows — For every entry in
components_diff.renamed, verify the heatmap has exactly ONE row for the renamed component (using current name), not TWO rows (one all-zero baseline, one all-zero current). - [ ] Heatmap anomaly detection executed — For every heatmap row with
baseline.Total > 0, current.Total == 0(disappeared) and every row withbaseline.Total == 0, current.Total > 0(appeared): verify that fingerprint cross-checking was performed. If a disappeared-appeared pair shares source files, class names, or namespace → it's a missed rename and must be reclassified. The heatmap should NOT have matching all-zero/all-new pairs with shared source files. - [ ] Comparison confidence score present —
diff-result.jsonmust containcomparison_confidencefield ("high" or "low"). If more than 3 unresolved heatmap anomalies exist → confidence must be "low" with warning banner in HTML. - [ ] Per-component STRIDE arithmetic — For each heatmap row:
S+T+R+I+D+E+A == TotalANDT1+T2+T3 == Totalfor both baseline and current. Any mismatch → FAIL. - [ ] Delta arrows match JSON data — For each heatmap cell,
delta = current - baseline. If delta == 0, no arrow. If delta > 0, ▲. If delta < 0, ▼. Spot-check at least 3 components. - [ ] Component removal source file verification — For every component in
components_diff.removed, verify itssource_filesare genuinely absent from the current commit. If source files still exist → reclassify as renamed or methodology gap.