AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

aks-troubleshooting

Debug live Azure Kubernetes Service (AKS) incidents with a read-only, evidence-first investigation. WHEN: pod crashes or Pending, CrashLoopBackOff, OOMKilled, ImagePullBackOff, node NotReady, DNS or ingress failure, connectivity timeout, network policy, SNAT exhaustion, node-pool scaling blocked by QuotaExceeded or InsufficientVCPUQuota, upgrade stuck, spot or zone disruption, a bare VMExtensionProvisioningError or AllocationFailed wrapper, an uncataloged capacity symptom, or 'investigate my AKS cluster'. DO NOT USE FOR: packet capture (use aks-network-capture); GPU or model-serving issues (use aks-gpu-inference); cluster creation or provisioning (use azure-kubernetes); cost (use cost-analysis from the optional azure-cost plugin); pod rightsizing (use azure-kubernetes); a fully qualified documented AKS signature with every required nested qualifier (use aks-known-issues); standalone failures on non-AKS Azure resources (use azure-diagnostics). Unqualified errors and open-ended incidents stay here only for AKS.

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--aks-troubleshooting at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 18

references/api-server-webhooks-tunnel.md

API server, admission webhook, and tunnel evidence

Bind every read to the named AKS cluster, affected object, nodes, and incident window. Use supplied evidence and approved host capabilities; unavailable telemetry is unknown, not absent.

API server pressure

For API latency, timeouts, HTTP 429, or "unable to handle the request," inspect Resource Health and AKS control-plane diagnostics when available, then correlate the affected command, recent bulk operations, and high-volume list/watch clients. Distinguish customer-driven request load from a platform condition. If the remaining evidence requires service-side control-plane access, stop and escalate with the cluster, timestamps, symptoms, correlation IDs, and completed read-only checks.

HTTP 429: managed API server guard versus client load

kubectl get flowschemas
kubectl get prioritylevelconfigurations
kubectl get flowschema aks-managed-apiserver-guard -o yaml
kubectl get events -n kube-system --field-selector type=Warning --sort-by=.lastTimestamp
  • A FlowSchema and PriorityLevelConfiguration named aks-managed-apiserver-guard mean AKS is throttling non-system client requests as a last-resort protection after repeated API-server OOM events. Its presence explains that requests are being throttled; it does not by itself identify the root cause, which is still excessive client load, oversized list/watch patterns, or etcd object growth. Report it as the throttling mechanism, keep reading evidence, and never delete or relabel the guard objects: that is a mutation the owner must approve after the load source is fixed.
  • Absence of the guard does not prove the client is at fault. Treat 429s without the guard as ordinary API Priority and Fairness throttling and identify the client from the request-volume evidence available to the customer (AKS control-plane diagnostics when configured, client-side logs, recent controller or CI rollouts).

etcd size and object count

# Metric name varies by Kubernetes version; the grep matches all three
kubectl get --raw /metrics | grep -E "etcd_db_total_size_in_bytes|apiserver_storage_size_bytes|apiserver_storage_db_total_size_in_bytes"
kubectl get --raw /metrics | grep -E '^apiserver_storage_objects' | sort -t' ' -k2 -nr | head -20
kubectl get events -A --sort-by=.lastTimestamp | tail -50
  • Learn describes an etcd database above 2 GB as large and 8 GB as the default size limit; more than roughly 10,000 objects of one type can slow etcd. Use the Azure portal's Diagnose and Solve Problems etcd capacity/performance tools for the breakdown when portal access is available.
  • A rising object count for one kind is correlation until the owning controller is evidenced (for example the resource's ownerReferences, managedFields manager, or a creating controller's logs). Name the kind and the suspected owner, quantify the growth, and stop. Do not delete objects; cleanup, retention, and ResourceQuota object quotas are owner decisions.
  • Free tier clusters have limited API-server and etcd resources and no HA; record the tier as evidence before attributing pressure to a client.

Admission webhooks

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o yaml
kubectl get service,endpoints,endpointslice -n <webhook-namespace>
kubectl get pods -n <webhook-namespace> -o wide

For the named webhook, inspect failurePolicy, timeoutSeconds, namespaceSelector, backing Service, endpoints, and ready pods. A failing webhook with failurePolicy: Fail can block matching operations, but an error string alone does not prove the backend cause or blast radius. Never delete, patch, or weaken a webhook until the backend state is proven and the owner approves the remediation.

Deployment Safeguards as an admission actor

If an admission denial or an unexpected pod mutation (CPU/memory requests or limits set or raised, pod anti-affinity or topology spread constraints added) does not match any customer ValidatingWebhookConfiguration or MutatingWebhookConfiguration, check whether AKS Deployment Safeguards is enabled at the Enforce level before attributing it to a customer webhook. Safeguards is enabled and set to Enforce by default on AKS Automatic clusters (namespaces can be excluded, but the level cannot be changed to Warn) and is optional Warn or Enforce on AKS Standard, so require the cluster's actual Safeguards state rather than partitioning on Automatic versus Standard:

az aks safeguards show -g <rg> -n <cluster>   # aks-preview extension: level, excluded namespaces, PSS level
kubectl get constraints                       # Gatekeeper constraints active on the cluster

Report the matching Safeguards policy or mutator by name and leave the enablement level, excluded namespaces, and workload manifest changes to the owner. Policy-by-policy manifest compatibility for AKS Automatic belongs to the nested azure-kubernetes Automatic readiness reference; hand off there when the question becomes "will this manifest be admitted."

Node-dependent logs, exec, and port-forward failures

If kubectl logs, exec, port-forward, or webhook calls fail only for pods on some nodes while ordinary API reads work, test the node-placement pattern and inspect the konnectivity-agent pods in kube-system. Port 10250 errors can indicate the API-server-to-kubelet tunnel path rather than pod networking. Inspect the affected nodes' NSG, firewall, UDR, and tunnel-component evidence read-only. Do not change node firewall or iptables state; escalate service-side or unsupported node conditions with the collected evidence.

Exact nested VMExtensionError_K8SAPIServerConnFail / CSE exit 51 belongs to aks-known-issues. Explicit packet capture belongs to aks-network-capture.

Sources: API server and etcd troubleshooting, tunnel connectivity, admission-controller triage, Deployment Safeguards, and AKS support policies.