AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

aks-troubleshooting

Debug live Azure Kubernetes Service (AKS) incidents with a read-only, evidence-first investigation. WHEN: pod crashes or Pending, CrashLoopBackOff, OOMKilled, ImagePullBackOff, node NotReady, DNS or ingress failure, connectivity timeout, network policy, SNAT exhaustion, node-pool scaling blocked by QuotaExceeded or InsufficientVCPUQuota, upgrade stuck, spot or zone disruption, a bare VMExtensionProvisioningError or AllocationFailed wrapper, an uncataloged capacity symptom, or 'investigate my AKS cluster'. DO NOT USE FOR: packet capture (use aks-network-capture); GPU or model-serving issues (use aks-gpu-inference); cluster creation or provisioning (use azure-kubernetes); cost (use cost-analysis from the optional azure-cost plugin); pod rightsizing (use azure-kubernetes); a fully qualified documented AKS signature with every required nested qualifier (use aks-known-issues); standalone failures on non-AKS Azure resources (use azure-diagnostics). Unqualified errors and open-ended incidents stay here only for AKS.

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--aks-troubleshooting at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 18

pod-failures.md

Pod Failures & Application Issues

Target-Bound Pod Evidence

Use the shipped collector instead of streaming raw logs into model context. It verifies the kube context against the named AKS resource before any Kubernetes API read, captures current and previous logs across all containers, and keeps the raw log files in a caller-selected artifact directory.

AKS_SUBSCRIPTION_ID=<subscription-id> \
  scripts/pod-deep-dive.sh \
  <namespace> <pod-name> <resource-group> <cluster-name> <kube-context> \
  <artifacts-directory>

The model-visible projection redacts credential-shaped values and is capped at 50 lines per current/previous log stream. Inspect expanded raw evidence only outside model context.


CrashLoopBackOff

Pod starts, crashes, restarts with exponential backoff (10s, 20s, 40s... up to 5m).

Diagnostics:

kubectl describe pod <pod-name> -n <namespace>
# Check: Exit Code, Reason, Last State, Events

Use the Previous Logs section from pod-deep-dive.sh; it preserves the last crashed container output without exposing the raw stream.

Decision tree:

| Exit Code | Meaning | Fix Path | | --------- | ----------------------------------------------------- | ------------------------------------------------------------- | | 0 | App exited successfully (unexpected for long-running) | Check if entrypoint/command is correct; app may be a one-shot | | 1 | Application error | Read logs - unhandled exception, missing config, bad startup | | 137 | OOMKilled (SIGKILL) | Increase resources.limits.memory; check for memory leaks | | 139 | Segfault (SIGSEGV) | Binary compatibility issue or native code bug | | 143 | SIGTERM - graceful shutdown | Pod was terminated; check if liveness probe killed it |

OOMKilled specifically:

kubectl describe pod <pod-name> -n <namespace> | grep -A2 "Last State"
# Reason: OOMKilled -> container exceeded memory limit

Fix: increase resources.limits.memory or optimize application memory usage. Check kubectl top pod <pod-name> -n <namespace> for actual usage.

OOM kill tracing with Inspektor Gadget: Use trace_oomkill (timeout 30) with --k8s-namespace <namespace> --k8s-podname <pod-name> to see which process was killed and memory at kill time. See references/inspektor-gadget.md.

Deep diagnostics with Inspektor Gadget (when logs and describe are inconclusive):

Use the IG base command pattern with --k8s-namespace <namespace> --k8s-podname <pod-name> and these gadgets:

  • trace_exec (timeout 30) — see what the container executes at startup
  • trace_open (timeout 30) — find missing configs/secrets (retval -2 = ENOENT, -13 = EACCES)
  • snapshot_process (timeout 5) — list running processes in the pod

See references/inspektor-gadget.md.


ImagePullBackOff

Pod can't pull the container image.

Diagnostics:

kubectl describe pod <pod-name> -n <namespace>
# Events section shows the exact pull error

| Error Message | Cause | Fix | | --------------------------------------- | ---------------------------- | -------------------------------------------------------------- | | ErrImagePull / ImagePullBackOff | Image name or tag is wrong | Verify image name and tag exist in the registry | | unauthorized: authentication required | Missing or wrong pull secret | Create/update imagePullSecrets on the pod or service account | | manifest unknown | Tag doesn't exist | Check available tags in the registry | | context deadline exceeded | Registry unreachable | Check network/firewall; for ACR, verify AKS -> ACR integration |

ACR integration check:

# Verify AKS is attached to ACR
az aks check-acr -g <rg> -n <cluster> --acr <acr-name>.azurecr.io

Pending Pods

Pod stays in Pending - scheduler can't place it.

Diagnostics:

kubectl describe pod <pod-name> -n <namespace>
# Events section shows why scheduling failed

| Event Message | Cause | Fix | | ---------------------------------------------------------------------- | ----------------------------------- | --------------------------------------------------------------- | | Insufficient cpu / Insufficient memory | No node has enough resources | Scale node pool; reduce resource requests; check for overcommit | | node(s) had taint ... that the pod didn't tolerate | Taint/toleration mismatch | Add matching toleration or use a different node pool | | node(s) didn't match Pod's node affinity/selector | Affinity rule unsatisfiable | Check nodeSelector or nodeAffinity rules | | persistentvolumeclaim ... not found / unbound | PVC not ready | Check PVC status; verify storage class exists | | 0/N nodes are available: N node(s) had volume node affinity conflict | Zonal disk vs pod in different zone | Use ZRS storage class or ensure same zone |


Readiness & Liveness Probe Failures

Readiness probe failure -> pod removed from Service endpoints (no traffic). Liveness probe failure -> pod killed and restarted.

Diagnostics:

kubectl describe pod <pod-name> -n <namespace>
# Look for: "Readiness probe failed" or "Liveness probe failed" in Events

# Check the pod's READY column - must show n/n
kubectl get pod <pod-name> -n <namespace>

| Symptom | Cause | Fix | | ------------------------------------ | ----------------------- | ---------------------------------------------------------- | | READY shows 0/1 but pod is Running | Readiness probe failing | Check probe path, port, and app health endpoint | | Pod restarts repeatedly | Liveness probe failing | Increase initialDelaySeconds; check if app starts slowly | | Probe timeout errors | App responds too slowly | Increase timeoutSeconds; check app performance |

💡 Tip: Set initialDelaySeconds on liveness probes to be longer than your app's startup time. A common mistake is killing pods before they finish initializing.


Resource Constraints (CPU/Memory)

Check actual usage vs limits:

kubectl top pod <pod-name> -n <namespace>
kubectl top pod -n <namespace> --sort-by=memory

# Compare with requests/limits
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].resources}'

| Symptom | Cause | Fix | | -------------------------------- | --------------------------------------- | --------------------------------------------------- | | OOMKilled (exit code 137) | Container exceeded memory limit | Increase limits.memory or fix memory leak | | CPU throttling (slow responses) | Container hitting CPU limit | Increase limits.cpu or remove CPU limits | | Pending - insufficient resources | Requests exceed available node capacity | Lower requests, scale nodes, or use larger VM sizes |

⚠️ Warning: Setting CPU limits can cause unnecessary throttling even when the node has spare capacity. Many teams set CPU requests but not limits. Memory limits should always be set.