references/orchestrator.md · 24000
are in External.
- Two runs on the same code MUST produce the same number of boundaries (±1). A difference of ≥2 boundaries is WRONG.
- NEVER create boundaries based on code layers (Presentation/Business/Data) — boundaries represent DEPLOYMENT trust boundaries, not code architecture
Boundary count locking:
- After identifying boundaries, LOCK the count. Two runs on the same code MUST produce the same number of boundaries (±1 acceptable if one run identifies an edge boundary the other doesn't)
- A 4-boundary vs 7-boundary difference on the same code is WRONG and indicates the naming rules were not followed
Additional naming rules:
- The SAME component must get the SAME
idregardless of which LLM model runs the analysis or how many times it runs - External actors (
Operator,AzureDataStudio, etc.) are ALWAYS included — never drop them - Datastores representing distinct storage (files, database) are ALWAYS separate components — never merge them
- Lock the component list before Step 2. Use these exact IDs in ALL subsequent files (architecture, DFD, STRIDE, findings, JSON)
- If two classes exist as separate files (e.g.,
ResponsesAPIService.csandLLMService.cs), they are TWO components even if they seem related
⛔ DATA FLOW COMPLETENESS (MANDATORY — ensures consistent flow enumeration across runs): Data flows MUST be enumerated exhaustively. Two independent analyses of the same codebase MUST produce the same set of flows. To achieve this:
⛔ RETURN FLOW MODELING RULE (addresses 24% variance in flow counts):
- DO NOT model separate return flows. A request-response pair is ONE bidirectional flow (use
<-->in Mermaid). - Example:
DF01: Operator <--> TUI(one flow for input and output) - Example:
DF03: MCPHost <--> HealthAgent(one flow for delegation and result) - DO model separate flows ONLY when the two directions use different protocols or semantics (e.g., HTTP request vs WebSocket push-back).
- Why: When runs independently decide whether to create 1 flow or 2 flows per interaction, the flow count varies by 20-30%. This rule eliminates that variance.
- Flow count formula:
# flows ≈ # unique component-to-component interactions. If component A talks to component B, that is 1 flow, not 2.
Flow completeness checklist (use <--> bidirectional flows per the return flow rule above):
- Ingress/reverse proxy flows:
DF_EndUser_to_NginxIngress(bidirectional<-->),DF_NginxIngress_to_Backend(bidirectional<-->). Each is ONE flow, not two. - Database/datastore flows:
DF_Service_to_Redis(bidirectional<-->). ONE flow per service-datastore pair. - Auth provider flows:
DF_Service_to_AzureAD(bidirectional<-->). ONE flow per service-auth pair. - Admin access flows:
DF_Operator_to_Service(bidirectional<-->). ONE per admin interaction. - Flow count locking: After enumerating flows, LOCK the count. Two runs on the same code MUST produce the same number of flows (±3 acceptable). A difference of >5 flows indicates incomplete enumeration.
⛔ EXTERNAL ENTITY INCLUSION RULES (addresses variance in which externals are modeled):
- ALWAYS include
AzureAD(orEntraID) as an external entity if the code acquires tokens from Azure AD / Microsoft Entra ID (look forChainedTokenCredential,ManagedIdentityCredential,AzureCliCredential, MSAL, or any OAuth2/OIDC flow). - ALWAYS include the infrastructure target (e.g.,
OnPremInfra,HCICluster) as an external entity if the code sends commands to external infrastructure via PowerShell, REST, or WMI. - ALWAYS include
AzureOpenAI(or equivalent LLM endpoint) if the code calls a cloud LLM API. - ALWAYS include
Operatoras an external actor for CLI/TUI tools, admin tools, or operator consoles. - Rule of thumb: If the code has a client class or config for a service, that service is an external entity.
⛔ TMT CATEGORY RULES (addresses category inconsistency across runs):
- Tool servers that expose APIs callable by agents →
SE.P.TMCore.WebSvc(NOTSE.P.TMCore.NetApp) - Network-level services that handle connections/sockets →
SE.P.TMCore.NetApp - Services that execute OS commands (PowerShell, bash) →
SE.P.TMCore.OSProcess - Services that store data to disk (SessionStore, FileLogger) →
SE.DS.TMCore.FS(classify as Data Store, NOT Process) - Rule: If a class's primary purpose is persisting data, it is a Data Store. If it does computation or orchestration, it is a Process. Never switch between runs.
⛔ DFD DIRECTION (MANDATORY — addresses layout variance):
- ALL DFDs MUST use
flowchart LR(left-to-right). NEVER useflowchart TB. - ALL summary DFDs MUST also use
flowchart LR. - This is immutable — do not change based on aesthetics or diagram shape.
Acronym rules for PascalCase:
- Preserve well-known acronyms as ALL-CAPS:
API,NFS,LLM,SQL,HCI,AD,UI,DB - Examples:
IngestionAPI(notIngestionApi),NFSServer(notNfsServer),AzureAD(notAzureAd),VectorDBAPI(notVectorDbApi) - Single-word technologies keep standard casing:
Redis,Milvus,PostgreSQL,Nginx
Common technology naming (use EXACTLY these IDs for well-known infrastructure):
- Redis cache/state:
Redis(neverDaprStateStore,RedisCache,StateStore) - Milvus vector DB:
Milvus(neverMilvusVectorDb,VectorDB) - NGINX ingress:
NginxIngress(neverIngressNginx) - Azure AD/Entra:
AzureAD(neverAzureAd,EntraID) - PostgreSQL:
PostgreSQL(neverPostgresDb,Postgres) - User/Operator:
Operatorfor admin users,EndUserfor end users - Azure OpenAI:
AzureOpenAI(neverOpenAIService,LLMEndpoint) - NFS:
NFSServer(neverNfsServer,FileShare) - If two LLM models are separate deployments, keep them separate (never merge
MistralLLM+PhiLLMintoLocalLlm)
BUT: for application-specific classes, use the EXACT class name from the code, NOT a technology label:
ResponsesAPIService.cs→ResponsesAPIService(NOTOpenAIService— the class IS named ResponsesAPIService)TaskProcessor.cs→TaskProcessor(NOTLocalLLM— the class IS named TaskProcessor)SessionStore.cs→SessionStore(NOTStatePersistence— the class IS named SessionStore) Component granularity rules (CRITICAL for stability):- Model components at the technology/service level, not the script/file level
- A Docker container running Kusto is
KustoContainer— NOT decomposed intoKustoService+IngestLogs+KustoDataDirectory - A Moby Docker engine is
MobyDockerEngine— NOTInstallMoby(the installer script is evidence, not the component) - An installer for a tool is
SetupInstaller— NOT renamed toInstallAzureEdgeDiagnosticTool(script filename) - Rule: if a component has one primary function (e.g., "run Kusto queries"), model it as ONE component regardless of how many scripts/files implement it
- Scripts are EVIDENCE for components, not components themselves
- Keep the same granularity across runs — never split a single component into sub-components or merge sub-components between runs
⛔ COMPONENT ID FORMAT (MANDATORY — addresses casing variance):
- ALL component IDs MUST be PascalCase. NEVER use kebab-case, snake_case, or camelCase.
- Examples:
HealthAgent(nothealth-agent),AzureAD(notazure-ad),MCPHost(notmcp-host) - This applies to ALL artifacts: 0.1-architecture.md, 1-threatmodel.md, DFD mermaid, STRIDE, findings, JSON.
⛔ STRIDE SCOPE RULE (addresses external entity analysis variance):
- STRIDE analysis in
2-stride-analysis.mdMUST include sections for ALL elements in the Element Table EXCEPT external actors (Operator, EndUser). - External services (AzureOpenAI, AzureAD, OnPremInfra) DO get STRIDE sections — they are attack surfaces from YOUR system's perspective.
- External actors (human users) do NOT get STRIDE sections — they are threat SOURCES, not targets.
- This means: if you have 20 elements total and 1 is an external actor, you write 19 STRIDE sections.
⛔ STRIDE DEPTH CONSISTENCY (addresses threat count variance):
- Each component MUST get ALL 7 STRIDE-A categories analyzed (S, T, R, I, D, E, A).
- Each STRIDE category MUST be explicitly addressed per component: either with one or more concrete threats, OR with an explicit
N/A — {1-sentence justification}row explaining why that category does not apply to this specific component. - A category may produce 0, 1, 2, 3, or more threats — the count depends on the component's actual attack surface. Do NOT cap at 1 threat per category. Components with rich security surfaces (API services, auth managers, command executors, LLM clients) should typically have 2-4 threats per relevant STRIDE category. Only simple components (static config, read-only data stores) should have mostly 0-1.
- Expected distribution: For a 15-component system: ~30% of STRIDE cells should be 0 (with N/A), ~40% should be 1, ~25% should be 2, ~5% should be 3+. If ALL cells are 0 or 1 (binary pattern) → the analysis is too shallow. Go back and identify additional threat vectors.
- N/A entries do NOT count toward threat totals in the Summary table. Only concrete threat rows count.
- The Summary table S/T/R/I/D/E/A columns show the COUNT of concrete threats per category (0 is valid if N/A was justified).
- This ensures comprehensive coverage while producing accurate, non-inflated threat counts.
-
Write architecture overview (
0.1-architecture.md)- Read
skeletons/skeleton-architecture.mdfirst — copy skeleton structure, fill[FILL]placeholders - System purpose, key components, top scenarios, tech stack, deployment
- Use the exact component IDs locked in Step 1 — do not rename or merge components
- Reference:
output-formats.mdfor template,diagram-conventions.mdfor architecture diagram styles
- Read
-
Inventory security infrastructure
- Identify security-enabling components before flagging gaps
- Reference:
analysis-principles.mdSecurity Infrastructure Inventory table
-
Produce threat model DFD (
1.1-threatmodel.mmd,1.2-threatmodel-summary.mmd,1-threatmodel.md)- Read
skeletons/skeleton-dfd.md,skeletons/skeleton-summary-dfd.md, andskeletons/skeleton-threatmodel.mdfirst - Reference:
diagram-conventions.mdfor DFD styles,tmt-element-taxonomy.mdfor element classification - ⚠️ BEFORE FINALIZING: Run the Pre-Render Checklist from
diagram-conventions.md
⛔ POST-DFD GATE — Run IMMEDIATELY after creating
1.1-threatmodel.mmd:- Count elements (nodes with
((...)),[(...),["..."]) in1.1-threatmodel.mmd - Count boundaries (
subgraphlines) - If elements > 15 OR boundaries > 4:
→ You MUST create
1.2-threatmodel-summary.mmdusingskeleton-summary-dfd.mdNOW → Do NOT proceed to1-threatmodel.mduntil the summary file exists - If threshold NOT met → skip summary, proceed to
1-threatmodel.md - Create
1-threatmodel.md(include Summary View section if summary was generated)
- Read
-
Enumerate threats per element and flow using STRIDE-A (
2-stride-analysis.md)- Read
skeletons/skeleton-stride-analysis.mdfirst — use Summary table and per-component structure - Reference:
analysis-principles.mdfor tier definitions,output-formats.mdfor STRIDE template - ⛔ PREREQUISITE FLOOR CHECK (per threat): Before assigning a prerequisite to any threat, look up the component's
Min PrerequisiteandDerived Tierin the Component Exposure Table (0.1-architecture.md). The threat's prerequisite MUST be ≥ the component's floor. The threat's tier MUST be ≥ the component's derived tier (i.e., if component is T2, no threat can be T1). Use the canonical prerequisite→tier mapping fromanalysis-principles.md.
- Read
-
For each threat: cite files/functions/endpoints, propose mitigations, provide verification steps
-
Verify findings — confirm each finding against actual configuration before documenting
- Reference:
analysis-principles.mdFinding Validation Checklist
- Reference:
7b. Technology sweep — Run the Technology-Specific Security Checklist from analysis-principles.md
- For every technology found in the repo (Redis, Milvus, PostgreSQL, Docker, K8s, ML models, LLMs, NFS, CI/CD, etc.), verify you have at least one finding or explicit mitigation
- This step catches gaps that component-level STRIDE misses (e.g., database auth defaults, container hardening, key management)
- Add any missing findings before proceeding to Step 8
-
Compile findings (
3-findings.md)- Reference:
output-formats.mdfor findings template and Related Threats link format - Reference:
skeletons/skeleton-findings.md— read this skeleton, copy VERBATIM, fill in[FILL]placeholders for each finding
⛔ PRE-WRITE GATE — Verify before calling
create_filefor3-findings.md:- Finding IDs:
### FIND-01:,### FIND-02:— sequential,FIND-prefix (NOTF01orF-01) - CVSS prefix: every vector starts with
CVSS:4.0/(NOT bareAV:N/AC:L/...) - Related Threats: each threat ID is a separate hyperlink
[TNN.X](2-stride-analysis.md#anchor)(NOT plain text) - Sub-sections:
#### Description,#### Evidence,#### Remediation,#### Verification(NOTRecommendation) - Sort: within each tier → Critical → Important → Moderate → Low → higher CVSS first
- All 10 mandatory attribute rows present per finding
- Deployment context gate (FAIL-CLOSED): Read
0.1-architecture.mdDeployment Classification and Component Exposure Table. If classification isLOCALHOST_DESKTOPorLOCALHOST_SERVICE:- ZERO findings may have
Exploitation Prerequisites=None→ fix toLocal Process Access(T2) orHost/OS Access(T3) - ZERO findings may be in
## Tier 1→ downgrade to T2/T3 based on prerequisite - ZERO CVSS vectors may use
AV:Nunless the specific component hasReachability = Externalin the Component Exposure Table → fix toAV:LFor ALL deployment classifications: - For EACH finding, look up its Component in the exposure table. The finding's prerequisite MUST be ≥ the component's
Min Prerequisite. The finding's tier MUST be ≥ the component'sDerived Tier. - Prerequisites MUST use only canonical values:
None,Authenticated User,Privileged User,Internal Network,Local Process Access,Host/OS Access,Admin Credentials,Physical Access,{Component} Compromise. ⛔Application AccessandHost Accessare FORBIDDEN. If ANY violation exists → DO NOT WRITE THE FILE. Fix all violations first.
- ZERO findings may have
⛔ Fail-fast gate: Immediately after writing, run the Inline Quick-Checks for
3-findings.mdfromverification-checklist.md. Fix before proceeding.⛔ MANDATORY: All 3 tier sections must be present. Even if a tier has zero findings, include the heading with a note:
## Tier 1 — Direct Exposure (No Prerequisites)→*No Tier 1 findings identified for this repository.*- This ensures structural consistency for comparison matching and validation.
⛔ COVERAGE VERIFICATION FEEDBACK LOOP (MANDATORY): After writing the Threat Coverage Verification table at the end of
3-findings.md:- Scan the table you just wrote. Count how many threats have status
✅ Coveredvs🔄 Mitigated by Platformvs⚠️ Needs Reviewvs⚠️ Accepted Risk. - If ANY threat has
⚠️ Accepted Risk→ FAIL. The tool cannot accept risks. Go back and create a finding for each one. - If Platform ratio > 20% → SUSPECT. Re-examine each
🔄 Mitigated by Platformentry: is the mitigation truly from an EXTERNAL system managed by a DIFFERENT team? If the mitigation is the repo's own code (auth middleware, file permissions, TLS config, localhost binding), reclassify asOpenand create a finding. - If ANY
Openthreat in2-stride-analysis.mdhas NO corresponding finding → create a finding NOW. Use the threat's description as the finding title, the mitigation column as the remediation guidance, and assign severity based on STRIDE category. - Update
3-findings.mdwith the newly created findings. Renumber sequentially. Update the Coverage table to show✅ Coveredfor each. - This loop is the ENTIRE POINT of the Coverage table — it's not documentation, it's a self-check that forces complete coverage. If you write the table and don't act on gaps, you've wasted the effort.
- Reference:
8b. Generate threat inventory (threat-inventory.json)
- Read
skeletons/skeleton-inventory.mdfirst — use exact field names and schema structure - After writing all markdown reports, compile a structured JSON inventory of all components, boundaries, data flows, threats, and findings - Use canonical PascalCase IDs for components (derived from class/file names) and keep display labels separate
- Use canonical flow IDs:
DF_{Source}_to_{Target}- Include identity keys on every threat and finding for future matching - Include deterministic identity fields for component and boundary matching across runs:- Component:
aliases,boundary_kind,fingerprint - Boundary:
kind,aliases,contains_fingerprint- Buildfingerprintfrom stable evidence (source files, endpoint neighbors, protocols, type) — never from prose wording - Normalize synonyms to the same canonical component ID (example:SupportAgentandSupportabilityAgent→SupportabilityAgent) and store alternate names inaliases- Sort arrays deterministically before writing JSON: componentsbyidboundariesbyidflowsbyidthreatsbyidthenidentity_key.component_idfindingsbyidthenidentity_key.component_id
- Component:
- Extract metrics (totals, per-tier counts, per-STRIDE-category counts)
- Include git metadata (commit SHA, branch, date) and analysis metadata (model, timestamps)
- Reference:
output-formats.mdfor thethreat-inventory.jsonschema - This file is NOT linked in 0-assessment.md but is always present in the output folder
⛔ PRE-WRITE SIZE CHECK (MANDATORY — before calling create_file for JSON):
Before writing threat-inventory.json, count the data you plan to include:
- Count total threats from
2-stride-analysis.md(grep^\| T\d+\.) - Count total findings from
3-findings.md(grep### FIND-) - Count total components from
0.1-architecture.md - If threats > 50 OR findings > 15: DO NOT use a single
create_filecall. Instead, use one of: (a) delegate to sub-agent, (b) Python extraction script, (c) chunked write strategy. - If threats ≤ 50 AND findings ≤ 15: single
create_fileis acceptable, but keep entries minimal (1-sentence description/mitigation fields).
⛔ POST-WRITE VALIDATION (MANDATORY — JSON Array Completeness):
After writing threat-inventory.json, immediately verify:
threats.length == metrics.total_threats— if mismatch, the threats array was truncated during generation. Rebuild by re-reading2-stride-analysis.mdand extracting every threat row.findings.length == metrics.total_findings— if mismatch, rebuild from3-findings.md.components.length == metrics.total_components— if mismatch, rebuild from architecture/element tables.
⛔ CROSS-FILE THREAT COUNT VERIFICATION (MANDATORY — catches dropped threats):
The JSON threats.length can match metrics.total_threats but BOTH can be wrong if threats were dropped during JSON generation. To catch this:
- Count threat rows in
2-stride-analysis.md: grep for^\| T\d+\.and count unique threat IDs - Compare this count to
threats.lengthin the JSON - If the markdown has MORE threats than the JSON → the JSON dropped threats. Rebuild the JSON by re-extracting ALL threats from
2-stride-analysis.md. - This is the #2 quality issue observed in testing (after truncation). Large repos (114+ threats) frequently have 1-3 threats dropped when sub-agents write the JSON from memory instead of re-reading the STRIDE file.
⛔ FIELD NAME COMPLIANCE GATE (MANDATORY — run immediately after array check): Read the first component and first threat from the JSON just written and verify these EXACT field names:
-
components[0]has key"display"(NOT"display_name", NOT"name") → if wrong, find-replace ALL occurrences -
threats[0]has key"stride_category"(NOT"category") → if wrong, find-replace ALL occurrences -
threats[0].identity_keyhas key"component_id"(threat→component link must be INSIDEidentity_key, NOT a top-levelcomponent_idfield on the threat) → if wrong, restructure -
threats[0]has BOTH"title"(short name, e.g., "Information Disclosure — Redis unencrypted traffic") AND"description"(longer prose). If onlydescriptionexists withouttitle, createtitlefrom the first sentence ofdescription. Ifnameorthreat_nameexists instead oftitle, find-replace totitle -
Why this matters: Downstream tooling depends on these exact field names. Wrong names cause zero-value heatmaps, broken component matching, and empty display labels in comparison reports.
-
If ANY field name is wrong: fix it NOW with find-replace on the JSON file before proceeding. Do NOT leave it for verification.
-
This is the #1 quality issue observed in testing. Large repos (20+ components, 80+ threats) frequently have truncated JSON arrays because the model runs out of output tokens. If ANY array is truncated, you MUST rebuild it before proceeding. Do NOT finalize with mismatched counts.
⛔ HARD GATE — TRUNCATION RECOVERY (MANDATORY): If post-write validation detects ANY array mismatch:
- DELETE the truncated
threat-inventory.jsonimmediately - DO NOT attempt to patch the truncated file — partial JSON is unreliable
- Regenerate using one of these strategies (in preference order):
a. Delegate to a sub-agent — hand the sub-agent the output folder path and instruct it to read
2-stride-analysis.mdand3-findings.md, then writethreat-inventory.json. The sub-agent has a fresh context window. b. Python extraction script — write a Python script that reads the markdown files, extracts threats/findings via regex, and writes the JSON. Run the script via terminal. c. Chunked write — use the Large Repo Strategy below. - Re-validate after regeneration — if still mismatched, repeat with the next strategy
- NEVER proceed to Step 9 (assessment) or Step 10 (verification) with mismatched counts
⛔ LARGE REPO STRATEGY (MANDATORY for repos with >60 threats): For repos producing more than ~60 threats, the JSON file can exceed output token limits if generated in one pass. Use this chunked approach:
- Write metadata + components + boundaries + flows + metrics first — these are small arrays
- Append threats in batches — write threats array with ~20 threats per append operation. Use
replace_string_in_fileto add batches to the existing file rather than writing the entire JSON in onecreate_filecall. - Append findings — similarly batch if >15 findings
- Final validation — read the completed file and verify all array lengths match metrics
Alternative approach: If chunked writing is not feasible, keep each threat/finding entry minimal:
descriptionfield: max 1 sentence (not full prose paragraphs)mitigationfield: