llm-trails: What LLMs reach for inside a breached AWS account

What it is

llm-trails is an MIT-licensed repo by Adan Alvarez, with research done by Claude and reviewed and directed by Alvarez. It is a companion to traildiscover.cloud. The README states no findings. The only figure it gives is an illustration of how to read rates: "85% read secrets" means when secrets are already located and readable, not in general. Results are in the dashboard and viz_data.json, which the page does not reproduce.

How it works

The repo has three folders and one data flow.

flowchart LR
  A["experiment/: run models, code answers"] -->|viz_data.py| B["data/viz_data.json"]
  B -->|fetched at load| C["website/index.html"]
  D["New runs"] --> A

• experiment/ holds the llmtrials harness, the prompts, the 13 situations, the codebook and the aggregation scripts. Its own experiment/README.md is the starting point for reproducing or extending the study.
• data/ holds the processed dataset, regenerated by the experiment scripts.
• website/ fetches data/ at load, so new data appears on the next load with no code change.

What to do

To view the dashboard locally, serve the repo root and open the site:

python -m http.server 8000
# then open http://localhost:8000/website/

To rerun the study and rebuild the dataset, follow experiment/README.md.

To publish on GitHub Pages, enable Settings → Pages → Build and deployment → Source: GitHub Actions. The workflow .github/workflows/pages.yml then deploys website/ with data/ copied in on every push to main.

Caveats

Everything is single-turn. The model states a plan, the harness records it, and nothing runs against a real account. Rates reflect the situation shown to the model, so they do not describe general behavior. The README does not list the 14 models, the 13 situations, the codebook categories or any per-model results, so the dashboard or dataset has to be opened to see them.