llm-trails: What LLMs reach for inside a breached AWS account

What it claims

llm-trails is an MIT-licensed defensive study that asks 14 models for their next move in 13 post-compromise AWS situations and maps each answer to CloudTrail events. The stated aim is to turn how models tend to respond into detection priorities, not to measure attack capability. The repo is a companion to traildiscover.cloud. The page credits the research to Claude, reviewed and directed by Adan Alvarez. The fetched page does not name the 14 models or list per-situation results. Those live in the dashboard and in data/viz_data.json.

How it works

The pipeline has three stages:

1. experiment/ runs the models through the llmtrials harness, codes the answers with the codebook, and aggregates them.
2. viz_data.py writes the processed dataset to data/viz_data.json.
3. website/index.html fetches that file at runtime and renders the dashboard.

Because the site reads data/ at load, new runs or added models appear in the dashboard with no code change. Reproduction and extension start from experiment/README.md.

How to use it

To view the dashboard locally, serve the repo root:

python -m http.server 8000
# then open http://localhost:8000/website/

For hosting, .github/workflows/pages.yml publishes website/ to GitHub Pages with data/ copied in alongside it. To enable it, go to Settings → Pages → Build and deployment and set Source to GitHub Actions. It then deploys on every push to main.

Limits

Every rate is conditional on its situation. The page's own example: "85% read secrets" means 85% when secrets are already located and readable, not 85% in general. Rates therefore cannot be compared across situations as general tendencies.

All answers are single-turn plans. Nothing runs against a real account, so the data shows stated intent, not what a model would do over a multi-step session with real tool feedback. The CloudTrail mapping comes from the codebook applied to text answers, not from observed API calls.