references/inspektor-gadget.md
Inspektor Gadget (IG) Reference
Use Inspektor Gadget for real-time, low-level node/pod diagnostics when kubectl is insufficient.
Limits
Linux nodes only; kernel 5.10+ with BTF; not Kata-sandboxed pods. Admission policy can deny privileged pods. If eBPF fails to load or a pod is denied, record the evidence as unavailable and do not relax policy.
Choose a path
- Discover (read-only):
kubectl get ds gadget -n gadget -o jsonpath='{.spec.template.spec.containers[0].image}'. The AKS extensionmicrosoft.inspektorgadgetalso creates this DaemonSet. - Deployed and
kubectl gadgetpresent (preferred; no privileged pod): requirekubectl auth can-i create pods/portforward -n gadget. Match the gadget tag tokubectl gadget version, because a mismatched version fails to load. Runkubectl --context <kube-context> gadget run <gadget>:<server-version> -n <ns> [-p <pod>] --timeout <s> -o json > <raw-artifact>.json. Never use--detach, which creates persistent instances. - Otherwise: use the debug-pod path below. Require
kubectl auth can-i create pods -n <debug-ns>and explicit approval. - Never install IG during an investigation (
kubectl gadget deploy, Helm, oraz k8s-extension create). Propose installation to the owner as a separate change.
If a check fails, name the identity and the missing verb, then stop that path.
Approved Image (debug-pod path)
mcr.microsoft.com/oss/v2/inspektor-gadget/ig:v0.51.0@sha256:6610863f6d8cae28800f9331756434639bca44be065719cbcfe76e34c91dffa4
Do not replace the digest with a tag-only reference. Re-review the upstream release and platform manifests before changing either the version or digest.
Base Command Pattern (debug-pod path)
The incident owner must explicitly approve privileged debug-pod creation and choose a finite outer deadline before this command is run:
IG_IMAGE='mcr.microsoft.com/oss/v2/inspektor-gadget/ig:v0.51.0@sha256:6610863f6d8cae28800f9331756434639bca44be065719cbcfe76e34c91dffa4'
timeout <approved-deadline> \
kubectl --context <kube-context> debug --profile=sysadmin \
node/<node-name> --attach --quiet --image="$IG_IMAGE" -- \
ig run <gadget>:v0.51.0 -o json --timeout <gadget-seconds> [filters...] \
> <raw-artifact>.json
Use both bounds: the outer deadline caps the whole kubectl debug operation, and the inner IG timeout caps the gadget. Use --timeout 5 for snapshot/top and --timeout 30 for trace/profile. Record the generated debug-pod name and delete that exact pod within the same approved operation.
Note: The gadget observes read-only state, but creating and deleting a
--profile=sysadmindebug pod are privileged cluster mutations. Approval and appropriate RBAC are mandatory.
Required: Resolve the node name first:
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.nodeName}'
Common Filters (debug-pod ig run; kubectl gadget uses -n/-p/-c)
| Filter | Description |
|---|---|
| --k8s-namespace <ns> | Scope to a Kubernetes namespace |
| --k8s-podname <pod> | Scope to a specific pod |
| --k8s-containername <ctr> | Scope to a specific container |
| --timeout <seconds> | Cap streaming duration for trace/profile gadgets |
| --max-entries <n> | Max entries per batch for top/profile gadgets |
| --map-fetch-interval <dur> | Map fetch interval for top (except top_process) and profile gadgets (default 1000ms) |
| --interval <dur> | Reporting interval for top_process only (e.g. 5s) |
| --syscall-filters <list> | Comma-separated syscalls for traceloop (e.g. open,connect,accept). Always specify to limit data volume |
Tip: For top/profile, set
--map-fetch-interval≤ half of--timeoutto collect at least one batch. E.g.--timeout 2 --map-fetch-interval 1000ms --max-entries 20.Note:
top_processuses--intervalinstead of--map-fetch-interval. E.g.--timeout 10 --interval 5s --max-entries 20.
Gadget Catalog
Networking
| Gadget | Type | What It Does | When To Use |
|---|---|---|---|
| trace_dns | trace | Trace DNS queries and responses with latency | DNS failures, NXDOMAIN, SERVFAIL, slow resolution, intermittent DNS |
| trace_tcp | trace | Trace TCP connect/accept/close events | Connection refused, timeouts, unexpected drops, mapping pod connectivity |
| trace_tcpretrans | trace | Trace TCP retransmissions | Network congestion, lossy links, high latency between pods/services |
| trace_bind | trace | Trace socket bind calls | Port conflicts, address-already-in-use errors |
| trace_sni | trace | Trace TLS SNI (Server Name Indication) values | HTTPS routing issues, ingress TLS debugging, mTLS problems |
| snapshot_socket | snapshot | List open sockets (TCP/UDP/Unix) | Port conflicts, listening ports, connection leaks, ECONNREFUSED |
| tcpdump | special | Capture raw packets in pcap-ng format | Deep packet inspection, protocol-level debugging, reproducing network issues |
tcpdump gadget
Keep raw pcap-ng outside model context:
IG_IMAGE='mcr.microsoft.com/oss/v2/inspektor-gadget/ig:v0.51.0@sha256:6610863f6d8cae28800f9331756434639bca44be065719cbcfe76e34c91dffa4'
timeout <approved-deadline> \
kubectl --context <kube-context> debug --profile=sysadmin \
node/<node-name> --attach --quiet --image="$IG_IMAGE" -- \
ig run tcpdump:v0.51.0 -o pcap-ng \
--k8s-namespace <ns> --k8s-podname <pod> \
--timeout 30 --pf "port 80" \
> <raw-artifact>.pcapng
Use --pf "<expr>" for a narrow tcpdump filter (for example, port 80 or host 10.0.0.1). Output must be -o pcap-ng (not -o json). Inspect the artifact with an approved offline packet-analysis tool rather than pasting packet payloads into model context.
Process & Workload
| Gadget | Type | What It Does | When To Use |
|---|---|---|---|
| snapshot_process | snapshot | List running processes in pod/node | PID pressure, unknown processes, verifying entrypoint, CrashLoopBackOff |
| trace_exec | trace | Trace process execution (execve calls) | CrashLoopBackOff (what actually runs), unexpected child processes, security audit |
| trace_oomkill | trace | Trace OOM kill events with victim details | OOMKilled pods — see which process was killed, memory usage at kill time |
| trace_signal | trace | Trace signals delivered to processes | Unexpected SIGKILL/SIGTERM, liveness probe kills, graceful shutdown issues |
| top_process | top | Rank processes by CPU/memory usage | Identifying resource-hungry processes inside a pod or across a node |
| profile_cpu | profile | CPU profiling via stack sampling | High CPU usage investigation, finding hot code paths |
| traceloop | trace | Record syscalls as a flight recorder | Catch-all for intermittent issues. Always use --syscall-filters (e.g., open,connect,accept) to limit data volume |
File & Storage
| Gadget | Type | What It Does | When To Use |
|---|---|---|---|
| trace_open | trace | Trace openat syscall | Missing config/secret files (ENOENT), permission denied (EACCES), startup failures |
| trace_fsslower | trace | Trace slow filesystem operations | Slow disk I/O, PVC performance issues, NFS/Azure Disk latency |
| top_file | top | Rank files by read/write activity | Identifying I/O-heavy files, noisy log writers, disk pressure diagnosis |
Security & Audit
| Gadget | Type | What It Does | When To Use |
|---|---|---|---|
| trace_capabilities | trace | Trace Linux capability checks | Permission denied from dropped capabilities, SecurityContext debugging |
Guardrails
- Prove the named AKS resource and kube context match before resolving the node or running IG.
- Discover an existing IG deployment and check permissions before choosing the debug-pod path.
- Require explicit approval for privileged debug-pod creation and deletion.
- Bound every run with the gadget timeout; on the debug-pod path also use only the digest-pinned image and an outer deadline.
- Scope every run to the symptom, namespace, pod, and supported filter set; do not use an unbounded catch-all trace.
- Keep raw JSON/pcap output outside model context and expose only a bounded, redacted finding summary.
- Confirm deletion of the exact generated debug pod on success, failure, interruption, or timeout.