AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

aks-troubleshooting

Debug live Azure Kubernetes Service (AKS) incidents with a read-only, evidence-first investigation. WHEN: pod crashes or Pending, CrashLoopBackOff, OOMKilled, ImagePullBackOff, node NotReady, DNS or ingress failure, connectivity timeout, network policy, SNAT exhaustion, node-pool scaling blocked by QuotaExceeded or InsufficientVCPUQuota, upgrade stuck, spot or zone disruption, a bare VMExtensionProvisioningError or AllocationFailed wrapper, an uncataloged capacity symptom, or 'investigate my AKS cluster'. DO NOT USE FOR: packet capture (use aks-network-capture); GPU or model-serving issues (use aks-gpu-inference); cluster creation or provisioning (use azure-kubernetes); cost (use cost-analysis from the optional azure-cost plugin); pod rightsizing (use azure-kubernetes); a fully qualified documented AKS signature with every required nested qualifier (use aks-known-issues); standalone failures on non-AKS Azure resources (use azure-diagnostics). Unqualified errors and open-ended incidents stay here only for AKS.

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read microsoft--aks-troubleshooting at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 18

references/inspektor-gadget.md

Inspektor Gadget (IG) Reference

Use Inspektor Gadget for real-time, low-level node/pod diagnostics when kubectl is insufficient.

Limits

Linux nodes only; kernel 5.10+ with BTF; not Kata-sandboxed pods. Admission policy can deny privileged pods. If eBPF fails to load or a pod is denied, record the evidence as unavailable and do not relax policy.

Choose a path

  1. Discover (read-only): kubectl get ds gadget -n gadget -o jsonpath='{.spec.template.spec.containers[0].image}'. The AKS extension microsoft.inspektorgadget also creates this DaemonSet.
  2. Deployed and kubectl gadget present (preferred; no privileged pod): require kubectl auth can-i create pods/portforward -n gadget. Match the gadget tag to kubectl gadget version, because a mismatched version fails to load. Run kubectl --context <kube-context> gadget run <gadget>:<server-version> -n <ns> [-p <pod>] --timeout <s> -o json > <raw-artifact>.json. Never use --detach, which creates persistent instances.
  3. Otherwise: use the debug-pod path below. Require kubectl auth can-i create pods -n <debug-ns> and explicit approval.
  4. Never install IG during an investigation (kubectl gadget deploy, Helm, or az k8s-extension create). Propose installation to the owner as a separate change.

If a check fails, name the identity and the missing verb, then stop that path.

Approved Image (debug-pod path)

mcr.microsoft.com/oss/v2/inspektor-gadget/ig:v0.51.0@sha256:6610863f6d8cae28800f9331756434639bca44be065719cbcfe76e34c91dffa4

Do not replace the digest with a tag-only reference. Re-review the upstream release and platform manifests before changing either the version or digest.

Base Command Pattern (debug-pod path)

The incident owner must explicitly approve privileged debug-pod creation and choose a finite outer deadline before this command is run:

IG_IMAGE='mcr.microsoft.com/oss/v2/inspektor-gadget/ig:v0.51.0@sha256:6610863f6d8cae28800f9331756434639bca44be065719cbcfe76e34c91dffa4'
timeout <approved-deadline> \
  kubectl --context <kube-context> debug --profile=sysadmin \
  node/<node-name> --attach --quiet --image="$IG_IMAGE" -- \
  ig run <gadget>:v0.51.0 -o json --timeout <gadget-seconds> [filters...] \
  > <raw-artifact>.json

Use both bounds: the outer deadline caps the whole kubectl debug operation, and the inner IG timeout caps the gadget. Use --timeout 5 for snapshot/top and --timeout 30 for trace/profile. Record the generated debug-pod name and delete that exact pod within the same approved operation.

Note: The gadget observes read-only state, but creating and deleting a --profile=sysadmin debug pod are privileged cluster mutations. Approval and appropriate RBAC are mandatory.

Required: Resolve the node name first:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.nodeName}'

Common Filters (debug-pod ig run; kubectl gadget uses -n/-p/-c)

| Filter | Description | |---|---| | --k8s-namespace <ns> | Scope to a Kubernetes namespace | | --k8s-podname <pod> | Scope to a specific pod | | --k8s-containername <ctr> | Scope to a specific container | | --timeout <seconds> | Cap streaming duration for trace/profile gadgets | | --max-entries <n> | Max entries per batch for top/profile gadgets | | --map-fetch-interval <dur> | Map fetch interval for top (except top_process) and profile gadgets (default 1000ms) | | --interval <dur> | Reporting interval for top_process only (e.g. 5s) | | --syscall-filters <list> | Comma-separated syscalls for traceloop (e.g. open,connect,accept). Always specify to limit data volume |

Tip: For top/profile, set --map-fetch-interval ≤ half of --timeout to collect at least one batch. E.g. --timeout 2 --map-fetch-interval 1000ms --max-entries 20.

Note: top_process uses --interval instead of --map-fetch-interval. E.g. --timeout 10 --interval 5s --max-entries 20.

Gadget Catalog

Networking

| Gadget | Type | What It Does | When To Use | |---|---|---|---| | trace_dns | trace | Trace DNS queries and responses with latency | DNS failures, NXDOMAIN, SERVFAIL, slow resolution, intermittent DNS | | trace_tcp | trace | Trace TCP connect/accept/close events | Connection refused, timeouts, unexpected drops, mapping pod connectivity | | trace_tcpretrans | trace | Trace TCP retransmissions | Network congestion, lossy links, high latency between pods/services | | trace_bind | trace | Trace socket bind calls | Port conflicts, address-already-in-use errors | | trace_sni | trace | Trace TLS SNI (Server Name Indication) values | HTTPS routing issues, ingress TLS debugging, mTLS problems | | snapshot_socket | snapshot | List open sockets (TCP/UDP/Unix) | Port conflicts, listening ports, connection leaks, ECONNREFUSED | | tcpdump | special | Capture raw packets in pcap-ng format | Deep packet inspection, protocol-level debugging, reproducing network issues |

tcpdump gadget

Keep raw pcap-ng outside model context:

IG_IMAGE='mcr.microsoft.com/oss/v2/inspektor-gadget/ig:v0.51.0@sha256:6610863f6d8cae28800f9331756434639bca44be065719cbcfe76e34c91dffa4'
timeout <approved-deadline> \
  kubectl --context <kube-context> debug --profile=sysadmin \
  node/<node-name> --attach --quiet --image="$IG_IMAGE" -- \
  ig run tcpdump:v0.51.0 -o pcap-ng \
  --k8s-namespace <ns> --k8s-podname <pod> \
  --timeout 30 --pf "port 80" \
  > <raw-artifact>.pcapng

Use --pf "<expr>" for a narrow tcpdump filter (for example, port 80 or host 10.0.0.1). Output must be -o pcap-ng (not -o json). Inspect the artifact with an approved offline packet-analysis tool rather than pasting packet payloads into model context.

Process & Workload

| Gadget | Type | What It Does | When To Use | |---|---|---|---| | snapshot_process | snapshot | List running processes in pod/node | PID pressure, unknown processes, verifying entrypoint, CrashLoopBackOff | | trace_exec | trace | Trace process execution (execve calls) | CrashLoopBackOff (what actually runs), unexpected child processes, security audit | | trace_oomkill | trace | Trace OOM kill events with victim details | OOMKilled pods — see which process was killed, memory usage at kill time | | trace_signal | trace | Trace signals delivered to processes | Unexpected SIGKILL/SIGTERM, liveness probe kills, graceful shutdown issues | | top_process | top | Rank processes by CPU/memory usage | Identifying resource-hungry processes inside a pod or across a node | | profile_cpu | profile | CPU profiling via stack sampling | High CPU usage investigation, finding hot code paths | | traceloop | trace | Record syscalls as a flight recorder | Catch-all for intermittent issues. Always use --syscall-filters (e.g., open,connect,accept) to limit data volume |

File & Storage

| Gadget | Type | What It Does | When To Use | |---|---|---|---| | trace_open | trace | Trace openat syscall | Missing config/secret files (ENOENT), permission denied (EACCES), startup failures | | trace_fsslower | trace | Trace slow filesystem operations | Slow disk I/O, PVC performance issues, NFS/Azure Disk latency | | top_file | top | Rank files by read/write activity | Identifying I/O-heavy files, noisy log writers, disk pressure diagnosis |

Security & Audit

| Gadget | Type | What It Does | When To Use | |---|---|---|---| | trace_capabilities | trace | Trace Linux capability checks | Permission denied from dropped capabilities, SecurityContext debugging |

Guardrails

  • Prove the named AKS resource and kube context match before resolving the node or running IG.
  • Discover an existing IG deployment and check permissions before choosing the debug-pod path.
  • Require explicit approval for privileged debug-pod creation and deletion.
  • Bound every run with the gadget timeout; on the debug-pod path also use only the digest-pinned image and an outer deadline.
  • Scope every run to the symptom, namespace, pod, and supported filter set; do not use an unbounded catch-all trace.
  • Keep raw JSON/pcap output outside model context and expose only a bounded, redacted finding summary.
  • Confirm deletion of the exact generated debug pod on success, failure, interruption, or timeout.