agent-egress-bench

What it is

agent-egress-bench (Agent Egress Bench / "aeb") is the actual repository behind the catalog summary. Beyond the concept, the page gives the concrete mechanics an engineer needs to run it against their own egress-mediation tool, plus the governance and comparability guarantees that determine whether a score means anything.

How it works

A result is only valid if two hand-offs outside the vendor's control both succeed. The reference runner stands up HTTP, TLS, WebSocket, DNS, and MCP fixtures and drives each case through your adapter; a declared route only authorizes an attempt — without proof of delivery and an observable verdict, the case is a "non-measurement," never a pass. Then aeb-validate reconciles, fully offline, every result row against its cited case, the summary against the rows, and the manifest against file digests, so a downloaded evidence bundle can be re-checked by someone who wasn't there.

Comparability rests on immutability: case IDs and semantics never change once merged (CI rejects edits against the merge base); a wrong verdict gets a new case plus a governance record, never a silent fix. Every result is scoped to the exact product, version, config, adapter, capability profile, corpus, and scoring version.

Preconditions / Affected

• Go 1.25 or newer — a distro golang package may be older; check go version first.
• The Pipelock reference adapter path additionally needs Linux, Git, Python 3, curl, jq, Make, tar, GNU timeout, a SHA-256 utility, and one of socat/ncat/nc.

What to do

Build the validator and check the corpus:

git clone --branch main https://github.com/luckyPipewrench/agent-egress-bench.git
cd agent-egress-bench
cd validate && go build -o aeb-validate . && cd ..
./validate/aeb-validate ./cases

Run the bundled Pipelock adapter (downloads the pinned release, verifies digest and version, confines it with Landlock/seccomp, starts fixtures, runs the corpus):

./scripts/run-pipelock-gauntlet.sh --doctor   # read-only preflight
./scripts/run-pipelock-gauntlet.sh

Use AEB_GO=/opt/go1.25/bin/go or --go to pick a toolchain outside PATH. In CI, pin the reusable Action to a commit; it fails unless the summary reports measurement_status: measured. A tagged release publishes a digest-pinned GHCR image with a signed runner-image.ref; offline: 'true' then runs under Docker network mode none (see docs/OCI-RUNNER.md).

Caveats

• The false-positive control group (cases/false-positive/) is scored on its own axis and never folded into the containment number — a block-everything tool scores perfectly on containment and is useless.
• Endpoint/process-boundary containment scenarios (detached descendants, direct network attempts) are scored separately and do not enter the Gauntlet wire score.
• PipeLab, the maintainer, also builds Pipelock, which ships as the reference adapter; the project publishes no ranking or mark and permits results unfavorable to Pipelock (see Governance).