AWS EKS Forensics: Data Sources and Investigation Tooling

What it is

A field guide from Synacktiv's CSIRT to what an EKS compromise leaves behind and how to pull it back out of AWS-native telemetry. The summary already names the three log layers and the general query targets; below are the concrete commands, detection tells, and constraints the summary cannot carry.

Preconditions

Compute mode decides node access and is the first triage step: standard clusters run worker nodes as EC2 instances reachable via SSH/SSM; under Auto Mode or Fargate the host is AWS-managed and direct node access is unavailable. Almost nothing is logged by default — control plane logs are off, Container Insights and GuardDuty Runtime Monitoring are opt-in — so the investigation's outcome is largely fixed before the incident.

What to do

Enable all five control-plane log types (all land in /aws/eks/[cluster]/cluster):

aws eks update-cluster-config --name <cluster> --region <region> \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

Static-pod persistence is the page's marquee detection. On EKS-optimized AMIs the kubelet does not watch a manifests dir by default (staticPodPath absent from /etc/kubernetes/kubelet/config.json), so an attacker must set it and restart the service — an artifact observable on the node before any pod event. The kubelet registers a mirror pod (visible to kubectl) and GuardDuty raises Execution:Kubernetes/AnomalousBehavior.WorkloadDeployed. The conclusive audit-log tell is a pod created by a system:node: identity carrying annotation kubernetes.io/config.source = "file" (vs "api"):

fields @timestamp, user.username, objectRef.name,
       `requestObject.metadata.annotations.kubernetes.io/config.source` as config_source
| filter verb = "create" and objectRef.resource = "pods"
| filter user.username like "system:node:"
| sort @timestamp desc

Node-local token theft — projected ServiceAccount tokens of pods scheduled on a node are a common target:

find /var/lib/kubelet/pods -path "*/kubernetes.io~projected/*/token" -type f

Node acquisition: prefer aws ssm start-session --target [node instance-id]. Fall back to kubectl debug node/[node] only as last resort — it schedules a workload on the node under examination and leaves fresh create pods entries in the audit log; use deliberately and document it.

Containment ordering matters: kubectl delete destroys process state, the writable upper layer, and projected tokens the analysis depends on, and a controller may reschedule the workload. Isolate at the network layer (NetworkPolicies, Security Groups) while keeping liveness/readiness probes satisfied so the pod is neither killed nor moved. But network isolation does not stop a stolen AWS identity — a node-role or ServiceAccount token replays from anywhere, so revoke/rotate those credentials.

Caveats

• GuardDuty is a black box: detection rules are undisclosed and non-extensible; a finding is a lead, never a verdict. Runtime Monitoring (eBPF syscall visibility) is optional, adds pods, and is unsupported on Fargate — without it, in-container command execution is the coverage gap that only a third-party agent fills.
• Kubelet checkpoint API (CRIU-based, Beta since v1.30) is not usable on EKS as of writing — the EKS-optimized AMIs do not ship CRIU in the CRI runtime.
• Third-party runtime tools (Falco, Sysdig, Wiz) are explicitly out of scope.