AWS EKS forensics: data sources and investigation tooling (Synacktiv)

What it is

Synacktiv CSIRT's EKS forensics guide, part of their Kubernetes Forensics series. Beyond the three-layer data-source map the summary names, the page carries verbatim CloudWatch Logs Insights queries, the enablement commands, node-level config paths, and an incident-response procedure (containment reflexes, acquisition ordering) the summary only gestures at.

Preconditions

EKS emits almost none of the relevant telemetry by default — control-plane logs are off, Container Insights and Runtime Monitoring are opt-in add-ons. Node access depends on compute mode: standard clusters run EC2 workers reachable via SSH/SSM; Auto Mode and Fargate give no direct node access, so establishing the mode is the first triage step. Runtime Monitoring (eBPF syscall visibility) does not support Fargate; without it, in-container command execution is a coverage gap.

What to do

Enable all five control-plane log types (single log group /aws/eks/[cluster]/cluster):

aws eks update-cluster-config --name <cluster> --region <region> \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

Detect a static-pod node-persistence — the pod is created by a system:node: identity and annotated kubernetes.io/config.source = "file" instead of "api":

fields @timestamp, user.username, objectRef.name,
       `requestObject.metadata.annotations.kubernetes.io/config.source` as config_source
| filter verb = "create" and objectRef.resource = "pods"
| filter user.username like "system:node:"
| sort @timestamp desc

On EKS-optimized AMIs the kubelet has no staticPodPath by default, so the attacker must first set it in /etc/kubernetes/kubelet/config.json and restart the service — an observable node artifact preceding the pod event.

Isolate stolen node-role activity in CloudTrail by excluding cluster egress:

fields @timestamp, eventName, sourceIPAddress, userAgent
| filter userIdentity.arn like "<node-role-name>"
| filter not (sourceIPAddress like /^10\.0\./) and sourceIPAddress != "<nat-gateway-eip>"
| stats count(*) as n by eventName, sourceIPAddress
| sort n desc

A burst of Describe*/List*/Get* from an off-cluster IP with an aws-cli user agent signals discovery from a stolen token.

Locate node-held projected ServiceAccount tokens for acquisition:

$ find /var/lib/kubelet/pods -path "*/kubernetes.io~projected/*/token" -type f

Response ordering that the summary omits:

• Do not kubectl delete — it discards process state, the writable upper layer, and projected tokens, and a controller may reschedule the workload. Isolate at the network level (NetworkPolicies, Security Groups) while keeping liveness/readiness probes satisfied so the pod is not killed or moved.
• Network isolation does not contain a stolen AWS identity — a token replays from anywhere. Rotate/revoke node-role credentials and leaked ServiceAccount tokens.
• Acquisition is a race against the pod lifecycle: live pod and node-local tokens first (volatile), centralised CloudWatch logs are durable and retrievable later.

Caveats

• kubectl debug node/[node] (mounts host under /host) is a last-resort access route — it schedules a workload on the node under examination and leaves fresh create pods audit entries.
• The kubelet checkpoint API (Beta, on by default since v1.30) relies on CRIU, which EKS-optimized AMIs do not ship, so container checkpointing is not usable on EKS as of writing.
• GuardDuty is a black box (rules undisclosed, not extensible); a finding is a lead to confirm, never a verdict. A stolen pod identity has no dedicated GuardDuty finding equivalent to the node-role InstanceCredentialExfiltration.