AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

aks-gpu-inference

Diagnose Day-2 AKS GPU and KAITO incidents using profile-aware, read-only evidence. WHEN: 'Insufficient nvidia.com/gpu', GPU pod Pending, model-load OOM, DCGM/VRAM, KAITO Workspace not ready, or GPU autoscaling. DO NOT USE FOR: setup (airunway-aks-setup), non-GPU incidents (aks-troubleshooting), standalone VM quota (azure-quotas), or generic cost (cost-analysis or cost-optimization from the optional azure-cost plugin).

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read microsoft--aks-gpu-inference at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 5

references/gpu-observability.md

GPU observability and model-load OOM

Bind telemetry to the affected cluster, namespace, pod, container, GPU, and incident window. Keep host/container-memory OOM and CUDA device-allocator OOM as separate live hypotheses. A Kubernetes OOM symptom alone does not prove GPU memory exhaustion, and a CUDA allocator error does not prove which workload condition exhausted the device. Exclude model-size calculations and GPU SKU-sizing tables, not container or host memory evidence.

Correlate the evidence

kubectl get pod -n <namespace> <pod> -o json
kubectl logs -n <namespace> <pod> -c <container> --previous --timestamps
kubectl get node <gpu-node> -o json
az aks nodepool show --subscription <subscription> -g <resource-group> \
  --cluster-name <cluster> -n <pool> --query gpuProfile -o json

For an observed managed profile (managementMode: Managed, driver: Install), AKS exposes its managed DCGM exporter on port 19400 and labels enabled nodes kubernetes.azure.com/dcgm-exporter=enabled. For an unmanaged or unknown profile, inspect the installed exporter's declared port; do not copy the managed port assumption.

Correlate the model-load timestamp and container termination with:

| Signal | Use | |---|---| | DCGM_FI_DEV_FB_USED | Frame-buffer memory consumed before failure | | DCGM_FI_DEV_FB_FREE | Remaining frame-buffer memory before failure | | DCGM_FI_DEV_GPU_UTIL | Kernel-active time; not an efficiency score | | Termination reason OOMKilled (often exit 137) | Supports a container-memory path; exit 137 alone is insufficient | | torch.cuda.OutOfMemoryError or CUDA allocation failure in model logs | Supports a device-allocation path; correlate with per-GPU incident-window telemetry |

If FB_USED rises while FB_FREE approaches exhaustion in the same GPU and incident window, the evidence supports VRAM pressure. Healthy aggregate frame-buffer values outside that GPU and incident window do not establish a cause. If historical telemetry is absent, keep device exhaustion unconfirmed. If the container was OOMKilled, inspect its memory limit/usage and node memory conditions rather than discarding the host-memory path.

When the user supplied evidence and prohibited commands, state that no reads ran. Require explicit authorization before monitoring mutation, restart, scaling, cordon/drain, or node-pool changes. Propose a target-bound telemetry collection plan instead.

Sources: managed exporter port, GPU observability, PyTorch CUDA OOM.