AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

aks-gpu-inference

Diagnose Day-2 AKS GPU and KAITO incidents using profile-aware, read-only evidence. WHEN: 'Insufficient nvidia.com/gpu', GPU pod Pending, model-load OOM, DCGM/VRAM, KAITO Workspace not ready, or GPU autoscaling. DO NOT USE FOR: setup (airunway-aks-setup), non-GPU incidents (aks-troubleshooting), standalone VM quota (azure-quotas), or generic cost (cost-analysis or cost-optimization from the optional azure-cost plugin).

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read microsoft--aks-gpu-inference at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 5

references/gpu-cost-and-scaling.md

GPU workload scaling versus node-pool scaling

Treat these as separate control loops.

| Loop | Scales | Evidence | |---|---|---| | KEDA with DCGM and Managed Prometheus | Workload replicas through an HPA | ScaledObject, HPA conditions, external metric, target workload | | AKS node scaling | GPU node-pool capacity | Observed gpuProfile, autoscaler setting, pool state, Pending demand |

For KEDA diagnosis, read without mutation:

kubectl get scaledobject -n <namespace> <name> -o yaml
kubectl describe hpa -n <namespace> <hpa>
kubectl get --raw \
  '/apis/external.metrics.k8s.io/v1beta1/namespaces/<namespace>/<metric>'

Confirm that the Prometheus query selects the intended workload and that the HPA reports AbleToScale and ScalingActive. KEDA responding to DCGM_FI_DEV_GPU_UTIL changes pod replicas; it does not prove that a GPU node can be provisioned.

Before discussing node capacity, read gpuProfile and enableAutoScaling. Managed GPU preview node pools do not support cluster autoscaler, so do not recommend cluster-autoscaler scale-to-zero for that profile. For driver-only or BYO pools, diagnose the observed node-scaling configuration separately from KEDA.

Do not enable KEDA, monitoring, or autoscaling, and do not scale a workload or pool, without explicit authorization. Route generic cost analysis to cost-analysis from the optional azure-cost plugin; if it is unavailable, direct the user to their client's supported plugin installation flow.

Source: KEDA and DCGM workload autoscaling.