AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

azure-diagnostics

Debug Azure production issues using AppLens, Azure Monitor, resource health, and safe triage, including evidence-only diagnosis without live access. WHEN: debug production issues, troubleshoot app service, app service high CPU, app service deployment failure, troubleshoot container apps, container app 503 or target port mismatch, troubleshoot functions, function telemetry missing or stopped after a config/connection-string change, troubleshoot AKS, VM RDP, Linux SSH, VM black screen, can't connect to VM, reset VM password, NSG or firewall blocking, kubectl cannot connect, kube-system/CoreDNS failures, pod pending, crashloop, node not ready, upgrade failures, analyze logs, KQL, insights, image pull failures, cold start issues, health probe failures, resource health, root cause of errors, troubleshoot event hubs, troubleshoot service bus, messaging SDK error, AMQP connection failure, message lock lost, lock renewal during long-running processing, service bus dead letter.

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--azure-diagnostics at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 42

troubleshooting/aks/node-issues.md

Node & Cluster Troubleshooting

Node NotReady

Diagnostics:

kubectl get nodes -o wide
kubectl describe node <node-name>
# Look for: Conditions, Taints, Events, resource usage, kubelet status

Condition decision tree:

| Condition | Value | Meaning | Fix Path | | -------------------- | ------- | --------------------------------- | ------------------------------------------------------------- | | Ready | False | kubelet stopped reporting | SSH to node; if unrecoverable, consider cordon/drain/delete* | | MemoryPressure | True | Node running out of memory | Evict pods; scale out pool; reduce pod density | | DiskPressure | True | OS disk or ephemeral storage full | Check logs and images; clean up or increase disk | | PIDPressure | True | Too many processes | App spawning excessive threads/processes; use IG snapshot_process | | NetworkUnavailable | True | CNI plugin issue | Check CNI pods in kube-system; node network config |

*Only after explicit user request for remediation and confirmation of workload impact.

AKS-specific - SSH to a node:

⚠️ Warning: kubectl debug node/... creates a privileged debug pod on the node and is not a read-only diagnostic step. Default to read-only evidence gathering first. Only suggest or run this after the user explicitly asks for remediation or approves a privileged diagnostic action and understands the change-control impact.

# Create a privileged debug pod on the node
kubectl debug node/<node-name> -it --image=mcr.microsoft.com/cbl-mariner/base/core:2.0

# Check kubelet status inside the node
chroot /host systemctl status kubelet
chroot /host journalctl -u kubelet -n 50

Optional remediation if kubelet can't recover (after confirmation): cordon -> drain -> delete. AKS auto-replaces via node pool VMSS.

⚠️ Warning: These commands are disruptive. By default, stay in read-only diagnostic mode. Only suggest or run them if the user has explicitly requested remediation and confirmed they understand the workload and PodDisruptionBudget impact.

kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
kubectl delete node <node-name>

Node Pool Not Scaling

Cluster Autoscaler Not Triggering

Diagnostics:

# Autoscaler logs
kubectl logs -n kube-system -l app=cluster-autoscaler --tail=100

# Autoscaler status
kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml

# Verify autoscaler is enabled on the node pool
az aks nodepool show -g <rg> --cluster-name <cluster> -n <nodepool> \
  --query "{autoscaleEnabled:enableAutoScaling, min:minCount, max:maxCount}"

Autoscaler won't scale up - common reasons:

  • Node pool already at maxCount
  • VM quota exhausted: az vm list-usage -l <region> -o table | grep -i "DSv3\|quota"
  • Pod nodeAffinity is unsatisfiable on any new node template
  • 10-minute cooldown period still active after last scale event

Autoscaler won't scale down - common reasons:

  • Pods with emptyDir local storage (configure --skip-nodes-with-local-storage=false if safe)
  • Standalone pods with no controller (not in a ReplicaSet)
  • cluster-autoscaler.kubernetes.io/safe-to-evict: "false" annotation on a pod

Manual Scaling

az aks nodepool scale -g <rg> --cluster-name <cluster> -n <nodepool> --node-count <n>

Resource Pressure & Capacity Planning

Check actual vs allocatable:

kubectl describe node <node> | grep -A6 "Allocated resources:"

See AKS resource reservations for allocatable math.

Ephemeral storage pressure:

# Check what's consuming ephemeral storage on a node
kubectl debug node/<node> -it --image=mcr.microsoft.com/cbl-mariner/base/core:2.0

Common culprit: high-volume container logs accumulating in /var/log/containers.

Deep diagnostics with Inspektor Gadget (PID pressure or unknown process load):

Use scripts/run-ig.sh --gadget snapshot_process --node <node-name> (or run-ig.ps1) to list all processes on the node. For node-wide scope, use --node (no pod filters). See references/inspektor-gadget.md.


Detailed Node And Cluster Guides

  • Upgrade Operations for node images, Kubernetes version upgrades, surge settings, and PDB-related drain blockers.
  • Spot And Zone Issues for spot evictions, tolerations, zone skew, and zonal storage or service behavior.