derailed/popeye

What it is

Popeye (derailed/popeye) is a read-only linter that inspects a live Kubernetes cluster and scores it. The page's operational details matter more than its feature list: how to run it, what it needs, and several traps in the defaults.

What to do

Run it locally against your kubeconfig. Without -n it uses the context namespace, or default, so it will not scan the whole cluster:

popeye -A
popeye -n fred
popeye --context olive
popeye -f spinach.yaml
popeye -n ns1 -s pod,svc --logs none

Run it in a container. --save inside the container writes to the container's /tmp, which is deleted by --rm, so map /tmp out:

docker run --rm -it \
  -v $HOME/.kube:/root/.kube \
  -e POPEYE_REPORT_DIR=/tmp/popeye \
  -v /tmp:/tmp \
  quay.io/derailed/popeye --context foo -n bar --save --output-file my_report.txt

Save a report to disk, or push it to S3 or Minio. Both S3 and Minio need AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the environment:

POPEYE_REPORT_DIR=$(pwd) popeye --save --out html --output-file report.html
popeye --s3-bucket s3://my-popeye/fred --s3-region us-west-2 --out json --save --output-file scan.json

Push Prometheus metrics. This goes through a Pushgateway, not a scrape endpoint:

popeye --push-gtwy-url http://localhost:9091

The published gauges are popeye_severity_total, popeye_code_total, popeye_linter_tally_total, popeye_report_errors_total and popeye_cluster_score. A sample Grafana dashboard is in the repo.

Run it in-cluster as a CronJob. The manifests are in the repo's k8s/ directory (kubectl apply -f k8s/popeye). The sample ClusterRole grants only get and list on:

• core resources, including secrets and configmaps
• apps, networking, batch and gateway resources
• autoscaling and policy resources
• RBAC resources
• metrics.k8s.io pods and nodes

Review that list against your policy, since it includes secrets cluster-wide.

Caveats

• Exit code: Popeye exits non-zero when it finds lint errors. In the CronJob, set --force-exit-zero, otherwise the pods end in an error state. The sample uses -o yaml --force-exit-zero.
• Schedule: The sample schedule "* */1 * * *" is commented as hourly, but as written it fires every minute for the first hour of each... in practice, check the cron expression before applying it. The page says hourly, but the minute field is *. 0 * * * * is the usual hourly form.
• Exclude regexes: Exclusion rules need the rx: prefix to be treated as regex, and a loose regex can hide more resources than intended. The page recommends an occasional unfiltered ("wide open") run to catch new issues.
• Exclude semantics: FQNs are namespace/name, and linter names are lowercase plural resource names (e.g. poddisruptionbudgets).
• Allocation checks: Over/under-allocation checks need a metrics-server.
• Registry allowlist: The registries list in spinach.yaml flags any resource pulling from a registry not on it. The sample allows quay.io and docker.io.
• Stability: The page calls the project brittle and work in progress. It says it works best with Kubernetes 1.25.x, may fail on older versions or with insufficient RBAC, and that the spinach format and metrics may change.
• Metrics cardinality: When scans are saved to disk and pushed, popeye_cluster_score gets an extra label for the artifact. The page itself notes this changes cardinality on every push.
• Terminal: It needs a 256-color terminal. Set TERM=xterm-256color on Unix if the output looks wrong.
• Scope: It covers only a curated set of resource types.

One correction to the schedule item above: the cron field is * */1 * * *, which means every minute during every hour, not once an hour. Despite the page's comment, that would run the job every minute, and concurrencyPolicy: Forbid only prevents overlap.