AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

aks-troubleshooting

Debug live Azure Kubernetes Service (AKS) incidents with a read-only, evidence-first investigation. WHEN: pod crashes or Pending, CrashLoopBackOff, OOMKilled, ImagePullBackOff, node NotReady, DNS or ingress failure, connectivity timeout, network policy, SNAT exhaustion, node-pool scaling blocked by QuotaExceeded or InsufficientVCPUQuota, upgrade stuck, spot or zone disruption, a bare VMExtensionProvisioningError or AllocationFailed wrapper, an uncataloged capacity symptom, or 'investigate my AKS cluster'. DO NOT USE FOR: packet capture (use aks-network-capture); GPU or model-serving issues (use aks-gpu-inference); cluster creation or provisioning (use azure-kubernetes); cost (use cost-analysis from the optional azure-cost plugin); pod rightsizing (use azure-kubernetes); a fully qualified documented AKS signature with every required nested qualifier (use aks-known-issues); standalone failures on non-AKS Azure resources (use azure-diagnostics). Unqualified errors and open-ended incidents stay here only for AKS.

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--aks-troubleshooting at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 18

node-issues.md

Node & Cluster Troubleshooting

Node NotReady — executable evidence first

Derive the cluster, subscription, resource group, and node from the user's selected target and the host's approved metadata capabilities. Stay within that scope; access available to a local Azure CLI identity is not permission to enumerate other subscriptions. If the target is unavailable or ambiguous, use the host's elicitation mechanism to ask the user to select or identify it. Do not require a local kubeconfig or an all-subscriptions sweep before asking.

Only where the host capability gate is satisfied, a scoped CLI read can resolve the cluster within an authorized subscription and resource group:

az aks list \
  --subscription <authorized-subscription-id> \
  --resource-group <authorized-resource-group> \
  --query "[].{id:id,name:name,resourceGroup:resourceGroup,fqdn:fqdn,privateFqdn:privateFqdn}" \
  -o json

For approved kubectl execution, verify that the selected context's API endpoint matches the selected AKS resource's fqdn or privateFqdn; do not assume the current context is the requested cluster. If no node was named, list non-Ready nodes only in that bound cluster through an approved host read or kubectl --context <verified-context> get nodes. Ask which node to investigate when the selection is ambiguous. Derive the VMSS name and instance ID from the selected node's providerID when available; otherwise request targeted metadata or operator-provided evidence rather than widening scope.

The following pipeline assumes that scope and context verification succeeded. If its execution capabilities are unavailable, request redacted equivalent outputs and record the missing evidence instead of running it:

AKS_SUBSCRIPTION_ID="<authorized-subscription-id>"
AKS_RG="<cluster-resource-group>"
AKS_NAME="<cluster-name>"
KUBE_CONTEXT="<verified-context>"
NODE="<node-name>"

# Kubernetes description, conditions, and node-scoped events
kubectl --context "$KUBE_CONTEXT" describe node "$NODE"
kubectl --context "$KUBE_CONTEXT" get node "$NODE" \
  -o jsonpath='{range .status.conditions[*]}{.lastTransitionTime}{"\t"}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'
kubectl --context "$KUBE_CONTEXT" get events --all-namespaces \
  --field-selector involvedObject.kind=Node,involvedObject.name="$NODE" \
  --sort-by='.metadata.creationTimestamp'

# AKS node resource group and Kubernetes node to VMSS instance mapping
NODE_RG=$(az aks show \
  --subscription "$AKS_SUBSCRIPTION_ID" \
  --resource-group "$AKS_RG" \
  --name "$AKS_NAME" \
  --query nodeResourceGroup -o tsv)
AGENT_POOL=$(kubectl --context "$KUBE_CONTEXT" get node "$NODE" \
  -o jsonpath='{.metadata.labels.agentpool}')
PROVIDER_ID=$(kubectl --context "$KUBE_CONTEXT" get node "$NODE" \
  -o jsonpath='{.spec.providerID}')
VMSS=$(printf '%s\n' "$PROVIDER_ID" |
  awk -F'/virtualMachineScaleSets/' '{print $2}' | cut -d/ -f1)
INSTANCE_ID=${PROVIDER_ID##*/}

printf 'nodeResourceGroup=%s\nagentPool=%s\nvmss=%s\ninstanceId=%s\n' \
  "$NODE_RG" "$AGENT_POOL" "$VMSS" "$INSTANCE_ID"

# AKS pool state
az aks nodepool show \
  --subscription "$AKS_SUBSCRIPTION_ID" \
  --resource-group "$AKS_RG" \
  --cluster-name "$AKS_NAME" \
  --name "$AGENT_POOL" \
  --query '{provisioningState:provisioningState,powerState:powerState.code,nodeImageVersion:nodeImageVersion,orchestratorVersion:orchestratorVersion}' \
  -o yaml

# Exact VMSS instance view and extension statuses
az vmss get-instance-view \
  --subscription "$AKS_SUBSCRIPTION_ID" \
  --resource-group "$NODE_RG" \
  --name "$VMSS" \
  --instance-id "$INSTANCE_ID" \
  -o json
az vmss get-instance-view \
  --subscription "$AKS_SUBSCRIPTION_ID" \
  --resource-group "$NODE_RG" \
  --name "$VMSS" \
  --instance-id "$INSTANCE_ID" \
  --query 'extensions[].{name:name,statuses:statuses,substatuses:substatuses}' \
  -o json

This block is the minimum Node NotReady evidence. It is incomplete until every command in it has produced output or the inability to collect it is recorded. Collect it before narrowing the failure to kubelet, host, provisioning, pressure, or network causes.

Privileged and service mutation boundary

Kubelet/service inspection through privileged node access requires explicit approval after the mandatory block identifies a node-local evidence gap. Restarting kubelet, cordoning, draining, deleting, reimaging, or replacing a node are separate remediations and require explicit approval with workload, PodDisruptionBudget, and change-control impact understood. None is part of the default evidence path.

Condition decision tree:

| Condition | Value | Evidence boundary | Next investigation | |---|---|---|---| | Ready | False | Compare the transition time and reason with VMSS instance and extension statuses | Determine whether the failure is kubelet, host, provisioning, or network related before requesting node access | | MemoryPressure | True | Review allocated resources, pod requests/limits, eviction events, and metrics | Identify the workload or node-pool capacity constraint | | DiskPressure | True | Review eviction events, pod ephemeral-storage requests/limits, and node image/OS state | Determine whether workload storage use or node storage capacity is responsible | | PIDPressure | True | Correlate condition transitions with workload placement and process evidence | Use IG snapshot_process if process-level evidence is required | | NetworkUnavailable | True | Review CNI pod state and logs plus node NIC routes and NSG evidence | Continue with Networking Troubleshooting |

Do not infer that a Ready=False condition requires a kubelet restart or node replacement. The Kubernetes condition reason, node events, VMSS provisioning state, and extension substatus determine the next branch.


Cluster or Node Pool in provisioningState: Failed

Read the failed operation record before reasoning from node or VMSS symptoms:

# 1. Operation record first. `az aks operation` is part of the aks-preview extension;
#    the CLI auto-installs it on first use, which may be disallowed on managed hosts.
az aks operation show-latest -g <rg> -n <cluster>
az aks operation show-latest -g <rg> -n <cluster> --nodepool-name <pool>
az aks operation show -g <rg> -n <cluster> --operation-id <id>

# 2. Activity-log fallback (documented Learn path; the only evidence if aks-preview is unavailable)
az monitor activity-log list -g <rg> --resource <cluster-resource-id> --status Failed --offset 24h -o table

# 3. Pool, scale set, and instance provisioning state
az aks nodepool show -g <rg> --cluster-name <cluster> -n <pool> --query '{state:provisioningState,power:powerState.code}'
az vmss show -g <node-rg> -n <vmss> --query provisioningState -o tsv
az vmss list-instances -g <node-rg> -n <vmss> --query '[].{id:instanceId,state:provisioningState}' -o table
  • The operation record carries the exact error code and message for the failing operation. An exact catalog error (for example the VMExtensionError_* codes) routes to aks-known-issues; quota errors follow the quota section below.
  • If the extension is missing and cannot be installed, say so and label the activity-log entry as degraded evidence: it identifies the failed operation and status message but is not the full operation record.
  • az resource update --ids <cluster-id> (reconcile) and az vmss update-instances are mutations. Report them as the documented recovery path only after the cause is understood and the owner approves.

Node Pool Not Scaling

Cluster Autoscaler Not Triggering

Confirm the precondition first: the cluster autoscaler scales up on Pending (unschedulable) pods, not on node CPU/memory pressure and not on HPA state. If the HPA is at maxReplicas and every pod is Running, the ceiling is the HPA, not the autoscaler; if kubectl get pods -A --field-selector=status.phase=Pending is empty, there is nothing for the autoscaler to act on.

Diagnostics:

# Autoscaler logs
kubectl logs -n kube-system -l app=cluster-autoscaler --tail=100

# Autoscaler status
kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml

# Verify autoscaler is enabled on the node pool
az aks nodepool show -g <rg> --cluster-name <cluster> -n <nodepool> \
  --query "{autoscaleEnabled:enableAutoScaling, min:minCount, max:maxCount}"

# HPA ceiling versus autoscaler: are any pods actually unschedulable?
kubectl get hpa -A
kubectl get pods -A --field-selector=status.phase=Pending

Platform metrics (Azure Monitor, Microsoft.ContainerService/managedClusters): cluster_autoscaler_unschedulable_pods_count (pods the autoscaler must place), cluster_autoscaler_cluster_safe_to_autoscale (0 means the autoscaler is refusing to act, for example after failed scale-ups or too many unready nodes), cluster_autoscaler_scale_down_in_cooldown, cluster_autoscaler_unneeded_nodes_count, and cluster_autoscaler_failed_scale_ups_total. Read them through an Azure Monitor capability or az monitor metrics list --resource <cluster-id> --metric <name>; Pending pods with safe_to_autoscale=0 is the autoscaler branch, Running pods with the HPA at max is not.

Autoscaler won't scale up - common reasons:

  • No Pending pods: the HPA is at maxReplicas, or requests fit on existing nodes
  • Node pool already at maxCount
  • VM quota exhausted (confirm the binding tier and operation evidence below)
  • Pod nodeAffinity is unsatisfiable on any new node template
  • A recent autoscaler decision is still governed by the cluster's configured autoscaler profile; compare the status ConfigMap timestamps and profile settings before concluding scaling is stuck

Quota evidence and owner action

Stay read-only by default. Do not conclude that quota is exhausted from an autoscaler symptom or a quota-usage listing alone. Require the failed activity or operation error plus the quota evidence exposed by that error for the same subscription and region. QuotaExceeded records provide Current Usage, Current Limit, and Additional Required; ErrCode_InsufficientVCPUQuota records provide requested and remaining vCPUs. Record the subscription ID, region, intended node-pool scale target, VM size and family, and other quota consumers. Report any missing item as an evidence gap.

Standard vCPU quota is enforced at two tiers per subscription and region: total regional vCPUs and VM-family vCPUs. Identify the binding tier from the exact error and usage evidence. Total Regional Cores identifies the regional tier, while a named family such as standardDSv5Family identifies the family tier. A scale operation must fit within both tiers.

If stating a minimum new limit for QuotaExceeded, calculate only Current Limit + Additional Required. Do not add unrequested headroom. Leave any larger request to the owner based on planned capacity.

For a binding quota limit, direct an authorized owner to the Azure portal Quotas flow for Compute and the affected region and VM family or regional tier. Submitting the request changes subscription quota state, so it requires owner approval and suitable subscription-scope permission. A request is reviewed; do not promise approval, timing, or retries, and do not submit quota changes by default.

After any approved quota change, re-read the AKS operation, node-pool state, and activity evidence before proposing a follow-up action. Do not assume the failed operation will or will not self-heal, and do not prescribe a retry without observing the post-approval state. Suggest another VM family, SKU, region, or scale target only after verifying support, quota, and capacity for the exact subscription and target; that choice belongs to the workload owner.

Sources: AKS QuotaExceeded troubleshooting, VM-family vCPU quota requests, and the az quota reference. The portal is preferred for requests; CLI create and update operations are writes and must remain owner-executed.

Autoscaler won't scale down - common reasons:

  • Pods with emptyDir local storage (configure --skip-nodes-with-local-storage=false if safe)
  • Standalone pods with no controller (not in a ReplicaSet)
  • cluster-autoscaler.kubernetes.io/safe-to-evict: "false" annotation on a pod

Manual Scaling

az aks nodepool scale -g <rg> --cluster-name <cluster> -n <nodepool> --node-count <n>

Resource Pressure & Capacity Planning

Separate node pressure, container limits, and scheduler capacity. A high current usage sample is not a leak, and a Pending pod is not proof of node pressure.

kubectl top node <node>
kubectl top pod -A --containers --sort-by=memory
kubectl get pods -A --field-selector spec.nodeName=<node> \
  -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,CPU_LIMIT:.spec.containers[*].resources.limits.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,MEM_LIMIT:.spec.containers[*].resources.limits.memory'
kubectl describe node <node>

| Signal | Meaning | |---|---| | MemoryPressure=True plus eviction events | Node-wide memory pressure; identify consumers and reservation/capacity before changing pod limits | | Container OOMKilled without node pressure | Container crossed its cgroup memory limit; compare the affected container, sidecars, and previous logs | | CPU usage near a container limit with throttling telemetry | Limit-induced throttling is plausible; CPU usage alone does not prove throttling | | Requests near allocatable but low live usage | Scheduler capacity constraint, not runtime pressure |

kubectl top requires Metrics Server and is a current sample, not historical evidence. If unavailable, use already-configured Azure Monitor or Prometheus telemetry for the incident window; do not enable monitoring during diagnosis. Use IG only after aggregate evidence identifies a node/pod scope.

See AKS resource reservations for allocatable math.

Ephemeral storage pressure:

# Correlate node pressure with workload requests, limits, placement, and events
kubectl describe node <node>
kubectl get pods -A --field-selector spec.nodeName=<node> \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,EPHEMERAL_REQUESTS:.spec.containers[*].resources.requests.ephemeral-storage,EPHEMERAL_LIMITS:.spec.containers[*].resources.limits.ephemeral-storage'
kubectl get events --all-namespaces \
  --field-selector involvedObject.kind=Node,involvedObject.name=<node> \
  --sort-by='.metadata.creationTimestamp'

If this evidence cannot identify the consumer, request approval before privileged node filesystem inspection.

Deep diagnostics with Inspektor Gadget (PID pressure or unknown process load):

Use snapshot_process (timeout 5) to list all processes on the node. For node-wide scope, omit pod filters. See references/inspektor-gadget.md.


Detailed Node And Cluster Guides

  • Upgrade Operations for node images, Kubernetes version upgrades, surge settings, and PDB-related drain blockers.
  • Auto-upgrade evidence when an expected automatic cluster or node OS upgrade did not happen.
  • Spot And Zone Issues for spot evictions, tolerations, zone skew, and zonal storage or service behavior.