references/azure-network-path.md
AKS Azure Network Path Evidence
Use after Kubernetes evidence places the failure outside the cluster data plane. Bind the target, failing five-tuple, and incident window first.
Resolve the source node NIC
AKS_SUBSCRIPTION_ID="<subscription-id>"
AKS_RG="<cluster-resource-group>"
AKS_NAME="<cluster-name>"
KUBE_CONTEXT="<verified-context>"
NS="<namespace>"
POD="<source-pod>"
DEST_IP="<destination-ip>"
DEST_PORT="<destination-port>"
PROTOCOL="TCP"
NODE_RG=$(az aks show --subscription "$AKS_SUBSCRIPTION_ID" \
-g "$AKS_RG" -n "$AKS_NAME" --query nodeResourceGroup -o tsv)
NODE=$(kubectl --context "$KUBE_CONTEXT" get pod "$POD" -n "$NS" \
-o jsonpath='{.spec.nodeName}')
NODE_IP=$(kubectl --context "$KUBE_CONTEXT" get node "$NODE" \
-o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')
PROVIDER_ID=$(kubectl --context "$KUBE_CONTEXT" get node "$NODE" \
-o jsonpath='{.spec.providerID}')
VMSS_NAME=$(printf '%s\n' "$PROVIDER_ID" |
awk -F'/virtualMachineScaleSets/' '{print $2}' | cut -d/ -f1)
INSTANCE_ID=${PROVIDER_ID##*/}
VMSS_VM_ID=${PROVIDER_ID#azure://}
NODE_NIC_ID=$(az vmss nic list-vm-nics \
--subscription "$AKS_SUBSCRIPTION_ID" \
-g "$NODE_RG" --vmss-name "$VMSS_NAME" --instance-id "$INSTANCE_ID" \
--query '[0].id' -o tsv)
NODE_SUBNET_ID=$(az network nic show --ids "$NODE_NIC_ID" \
--query 'ipConfigurations[0].subnet.id' -o tsv)
POOL=$(kubectl --context "$KUBE_CONTEXT" get node "$NODE" \
-o jsonpath='{.metadata.labels.agentpool}')
POD_SUBNET_ID=$(az aks nodepool show --subscription "$AKS_SUBSCRIPTION_ID" \
-g "$AKS_RG" --cluster-name "$AKS_NAME" -n "$POOL" \
--query podSubnetId -o tsv)
SOURCE_SUBNET_ID=${POD_SUBNET_ID:-$NODE_SUBNET_ID}
az vmss nic list-vm-nics is Uniform-VMSS-specific. For Flexible VMSS, use
az network nic. Never guess among multiple NICs or IP configurations.
Prove NSG and route decisions
az network nic list-effective-nsg --ids "$NODE_NIC_ID" -o json
az network nic show-effective-route-table --ids "$NODE_NIC_ID" -o json
az network vnet subnet show --ids "$SOURCE_SUBNET_ID" \
--query '{nsg:networkSecurityGroup.id,routeTable:routeTable.id,natGateway:natGateway.id,serviceEndpoints:serviceEndpoints}' \
-o yaml
# Network Watcher models the VMSS instance/node NIC, not Kubernetes
# NetworkPolicy or every CNI pod path. Use the node NIC's private IP.
az network watcher show-next-hop \
--vm "$VMSS_VM_ID" --nic "$NODE_NIC_ID" \
--source-ip "$NODE_IP" --dest-ip "$DEST_IP" -o json
az network watcher test-ip-flow \
--vm "$VMSS_VM_ID" --nic "$NODE_NIC_ID" \
--direction Outbound --protocol "$PROTOCOL" \
--local "$NODE_IP:*" --remote "$DEST_IP:$DEST_PORT" -o json
The CLI requires --vm even with --nic. If Network Watcher rejects the VMSS
instance or is unavailable, use effective NIC evidence and record the gap. Its
Allow result does not clear NetworkPolicy, CNI, firewall/NVA policy,
destination ACLs, or the application.
Interpret the route chosen by longest-prefix match:
| Effective next hop | Continue with |
|---|---|
| None | Identify the active blackhole UDR or invalid route |
| VirtualAppliance | Resolve Azure Firewall versus NVA; inspect policy, health, logs, and onward route |
| VirtualNetworkGateway | Verify gateway/BGP routes; do not assume transit |
| VNetPeering | Inspect both peering objects and the return path |
| Internet | Resolve the actual SNAT identity for the cluster outbound type |
For an NVA, verify enableIPForwarding: true; appliance commands need separate
authorization.
Egress and asymmetric routing
az aks show --subscription "$AKS_SUBSCRIPTION_ID" -g "$AKS_RG" -n "$AKS_NAME" \
--query '{outboundType:networkProfile.outboundType,effectiveOutboundIPs:networkProfile.loadBalancerProfile.effectiveOutboundIPs[].id}' \
-o yaml
az network vnet subnet show --ids "$SOURCE_SUBNET_ID" \
--query '{natGateway:natGateway.id,routeTable:routeTable.id}' -o yaml
| Outbound evidence | Diagnostic rule |
|---|---|
| Subnet NAT gateway | Inspect its public identity and incident-window DroppedSNATConnections |
| loadBalancer | Inspect managed LB outbound rules and effective outbound IPs; node count is not exhaustion evidence |
| userDefinedRouting or none | Follow 0.0.0.0/0 and prove the next hop's SNAT identity |
| block | Public egress is intentionally blocked except for evidenced paths |
For LB/Application Gateway inbound traffic, compare the backend return route toward the client. A UDR through a stateful firewall/NVA can make the path asymmetric. Require forward, return, and stateful-hop evidence.
Private endpoints and DNS
Preserve the pod's CNAME/A response and compare it with the endpoint NIC:
kubectl --context "$KUBE_CONTEXT" exec "$POD" -n "$NS" -- \
nslookup <service-fqdn>
az network private-endpoint show --ids <private-endpoint-id> \
--query '{state:provisioningState,subnet:subnet.id,nics:networkInterfaces[].id,connections:privateLinkServiceConnections[].privateLinkServiceConnectionState.status}' \
-o yaml
az network private-endpoint dns-zone-group list \
-g <private-endpoint-resource-group> --endpoint-name <private-endpoint-name> \
-o json
az network private-dns record-set a list \
-g <private-dns-resource-group> -z <privatelink-zone> -o table
az network private-dns link vnet list \
-g <private-dns-resource-group> -z <privatelink-zone> -o table
- The FQDN must reach the expected
privatelink.*name and endpoint IP. - With custom VNet DNS, inspect the server/Private Resolver forwarding path. A zone link alone does not prove the pod's configured resolver can query it.
- For a private cluster, apply the same chain to
privateFqdn; keep client-to-API DNS separate from pod-to-service DNS. - An approved endpoint with correct DNS but failed TCP moves the investigation to NSG, UDR, peering, destination policy, and application evidence.
Service endpoints
A service endpoint keeps public DNS. Require a Succeeded endpoint on the
source subnet and a target-service firewall/VNet rule for that exact subnet.
It is not transitive from another VNet or on-premises.
VNet peering and hub-spoke
az network vnet peering show -g <source-vnet-rg> \
--vnet-name <source-vnet> -n <source-to-remote-peering> \
--query '{state:peeringState,access:allowVirtualNetworkAccess,forwarded:allowForwardedTraffic,gatewayTransit:allowGatewayTransit,useRemoteGateways:useRemoteGateways,remote:remoteVirtualNetwork.id}' \
-o yaml
az network vnet peering show -g <remote-vnet-rg> \
--vnet-name <remote-vnet> -n <remote-to-source-peering> \
--query '{state:peeringState,access:allowVirtualNetworkAccess,forwarded:allowForwardedTraffic,gatewayTransit:allowGatewayTransit,useRemoteGateways:useRemoteGateways,remote:remoteVirtualNetwork.id}' \
-o yaml
- Both directions must be
Connected; verify address spaces and routes. - Peering is not transitive. Spoke-to-spoke traffic needs an evidenced routed path, commonly UDRs through a hub firewall/NVA.
- Gateway transit pairs hub
allowGatewayTransitwith spokeuseRemoteGateways; appliance transit also needs forwarded traffic and symmetric routes.
Sources: AKS outbound types, Network Watcher next hop, IP flow verify, private endpoint DNS, and VNet peering.