AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

aks-troubleshooting

Debug live Azure Kubernetes Service (AKS) incidents with a read-only, evidence-first investigation. WHEN: pod crashes or Pending, CrashLoopBackOff, OOMKilled, ImagePullBackOff, node NotReady, DNS or ingress failure, connectivity timeout, network policy, SNAT exhaustion, node-pool scaling blocked by QuotaExceeded or InsufficientVCPUQuota, upgrade stuck, spot or zone disruption, a bare VMExtensionProvisioningError or AllocationFailed wrapper, an uncataloged capacity symptom, or 'investigate my AKS cluster'. DO NOT USE FOR: packet capture (use aks-network-capture); GPU or model-serving issues (use aks-gpu-inference); cluster creation or provisioning (use azure-kubernetes); cost (use cost-analysis from the optional azure-cost plugin); pod rightsizing (use azure-kubernetes); a fully qualified documented AKS signature with every required nested qualifier (use aks-known-issues); standalone failures on non-AKS Azure resources (use azure-diagnostics). Unqualified errors and open-ended incidents stay here only for AKS.

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--aks-troubleshooting at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 18

references/azure-network-path.md

AKS Azure Network Path Evidence

Use after Kubernetes evidence places the failure outside the cluster data plane. Bind the target, failing five-tuple, and incident window first.

Resolve the source node NIC

AKS_SUBSCRIPTION_ID="<subscription-id>"
AKS_RG="<cluster-resource-group>"
AKS_NAME="<cluster-name>"
KUBE_CONTEXT="<verified-context>"
NS="<namespace>"
POD="<source-pod>"
DEST_IP="<destination-ip>"
DEST_PORT="<destination-port>"
PROTOCOL="TCP"

NODE_RG=$(az aks show --subscription "$AKS_SUBSCRIPTION_ID" \
  -g "$AKS_RG" -n "$AKS_NAME" --query nodeResourceGroup -o tsv)
NODE=$(kubectl --context "$KUBE_CONTEXT" get pod "$POD" -n "$NS" \
  -o jsonpath='{.spec.nodeName}')
NODE_IP=$(kubectl --context "$KUBE_CONTEXT" get node "$NODE" \
  -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')
PROVIDER_ID=$(kubectl --context "$KUBE_CONTEXT" get node "$NODE" \
  -o jsonpath='{.spec.providerID}')
VMSS_NAME=$(printf '%s\n' "$PROVIDER_ID" |
  awk -F'/virtualMachineScaleSets/' '{print $2}' | cut -d/ -f1)
INSTANCE_ID=${PROVIDER_ID##*/}
VMSS_VM_ID=${PROVIDER_ID#azure://}
NODE_NIC_ID=$(az vmss nic list-vm-nics \
  --subscription "$AKS_SUBSCRIPTION_ID" \
  -g "$NODE_RG" --vmss-name "$VMSS_NAME" --instance-id "$INSTANCE_ID" \
  --query '[0].id' -o tsv)
NODE_SUBNET_ID=$(az network nic show --ids "$NODE_NIC_ID" \
  --query 'ipConfigurations[0].subnet.id' -o tsv)
POOL=$(kubectl --context "$KUBE_CONTEXT" get node "$NODE" \
  -o jsonpath='{.metadata.labels.agentpool}')
POD_SUBNET_ID=$(az aks nodepool show --subscription "$AKS_SUBSCRIPTION_ID" \
  -g "$AKS_RG" --cluster-name "$AKS_NAME" -n "$POOL" \
  --query podSubnetId -o tsv)
SOURCE_SUBNET_ID=${POD_SUBNET_ID:-$NODE_SUBNET_ID}

az vmss nic list-vm-nics is Uniform-VMSS-specific. For Flexible VMSS, use az network nic. Never guess among multiple NICs or IP configurations.

Prove NSG and route decisions

az network nic list-effective-nsg --ids "$NODE_NIC_ID" -o json
az network nic show-effective-route-table --ids "$NODE_NIC_ID" -o json
az network vnet subnet show --ids "$SOURCE_SUBNET_ID" \
  --query '{nsg:networkSecurityGroup.id,routeTable:routeTable.id,natGateway:natGateway.id,serviceEndpoints:serviceEndpoints}' \
  -o yaml

# Network Watcher models the VMSS instance/node NIC, not Kubernetes
# NetworkPolicy or every CNI pod path. Use the node NIC's private IP.
az network watcher show-next-hop \
  --vm "$VMSS_VM_ID" --nic "$NODE_NIC_ID" \
  --source-ip "$NODE_IP" --dest-ip "$DEST_IP" -o json
az network watcher test-ip-flow \
  --vm "$VMSS_VM_ID" --nic "$NODE_NIC_ID" \
  --direction Outbound --protocol "$PROTOCOL" \
  --local "$NODE_IP:*" --remote "$DEST_IP:$DEST_PORT" -o json

The CLI requires --vm even with --nic. If Network Watcher rejects the VMSS instance or is unavailable, use effective NIC evidence and record the gap. Its Allow result does not clear NetworkPolicy, CNI, firewall/NVA policy, destination ACLs, or the application.

Interpret the route chosen by longest-prefix match:

| Effective next hop | Continue with | |---|---| | None | Identify the active blackhole UDR or invalid route | | VirtualAppliance | Resolve Azure Firewall versus NVA; inspect policy, health, logs, and onward route | | VirtualNetworkGateway | Verify gateway/BGP routes; do not assume transit | | VNetPeering | Inspect both peering objects and the return path | | Internet | Resolve the actual SNAT identity for the cluster outbound type |

For an NVA, verify enableIPForwarding: true; appliance commands need separate authorization.

Egress and asymmetric routing

az aks show --subscription "$AKS_SUBSCRIPTION_ID" -g "$AKS_RG" -n "$AKS_NAME" \
  --query '{outboundType:networkProfile.outboundType,effectiveOutboundIPs:networkProfile.loadBalancerProfile.effectiveOutboundIPs[].id}' \
  -o yaml
az network vnet subnet show --ids "$SOURCE_SUBNET_ID" \
  --query '{natGateway:natGateway.id,routeTable:routeTable.id}' -o yaml

| Outbound evidence | Diagnostic rule | |---|---| | Subnet NAT gateway | Inspect its public identity and incident-window DroppedSNATConnections | | loadBalancer | Inspect managed LB outbound rules and effective outbound IPs; node count is not exhaustion evidence | | userDefinedRouting or none | Follow 0.0.0.0/0 and prove the next hop's SNAT identity | | block | Public egress is intentionally blocked except for evidenced paths |

For LB/Application Gateway inbound traffic, compare the backend return route toward the client. A UDR through a stateful firewall/NVA can make the path asymmetric. Require forward, return, and stateful-hop evidence.

Private endpoints and DNS

Preserve the pod's CNAME/A response and compare it with the endpoint NIC:

kubectl --context "$KUBE_CONTEXT" exec "$POD" -n "$NS" -- \
  nslookup <service-fqdn>
az network private-endpoint show --ids <private-endpoint-id> \
  --query '{state:provisioningState,subnet:subnet.id,nics:networkInterfaces[].id,connections:privateLinkServiceConnections[].privateLinkServiceConnectionState.status}' \
  -o yaml
az network private-endpoint dns-zone-group list \
  -g <private-endpoint-resource-group> --endpoint-name <private-endpoint-name> \
  -o json
az network private-dns record-set a list \
  -g <private-dns-resource-group> -z <privatelink-zone> -o table
az network private-dns link vnet list \
  -g <private-dns-resource-group> -z <privatelink-zone> -o table
  • The FQDN must reach the expected privatelink.* name and endpoint IP.
  • With custom VNet DNS, inspect the server/Private Resolver forwarding path. A zone link alone does not prove the pod's configured resolver can query it.
  • For a private cluster, apply the same chain to privateFqdn; keep client-to-API DNS separate from pod-to-service DNS.
  • An approved endpoint with correct DNS but failed TCP moves the investigation to NSG, UDR, peering, destination policy, and application evidence.

Service endpoints

A service endpoint keeps public DNS. Require a Succeeded endpoint on the source subnet and a target-service firewall/VNet rule for that exact subnet. It is not transitive from another VNet or on-premises.

VNet peering and hub-spoke

az network vnet peering show -g <source-vnet-rg> \
  --vnet-name <source-vnet> -n <source-to-remote-peering> \
  --query '{state:peeringState,access:allowVirtualNetworkAccess,forwarded:allowForwardedTraffic,gatewayTransit:allowGatewayTransit,useRemoteGateways:useRemoteGateways,remote:remoteVirtualNetwork.id}' \
  -o yaml
az network vnet peering show -g <remote-vnet-rg> \
  --vnet-name <remote-vnet> -n <remote-to-source-peering> \
  --query '{state:peeringState,access:allowVirtualNetworkAccess,forwarded:allowForwardedTraffic,gatewayTransit:allowGatewayTransit,useRemoteGateways:useRemoteGateways,remote:remoteVirtualNetwork.id}' \
  -o yaml
  • Both directions must be Connected; verify address spaces and routes.
  • Peering is not transitive. Spoke-to-spoke traffic needs an evidenced routed path, commonly UDRs through a hub firewall/NVA.
  • Gateway transit pairs hub allowGatewayTransit with spoke useRemoteGateways; appliance transit also needs forwarded traffic and symmetric routes.

Sources: AKS outbound types, Network Watcher next hop, IP flow verify, private endpoint DNS, and VNet peering.