Tracekit: Tamper-Evident Intent-Reasoning-Action Auditing for Autonomous Coding Agents

What it claims

Tracekit records a coding agent's session in three channels: the human's prompt, the model's stated reasoning, and the tool calls it actually ran. These go into a SHA-256 hash-chained JSONL ledger, and the system flags places where the channels disagree. The authors treat the model's self-report as testimony to be checked against actions, not as a view into the model's internals. The implementation is about 2,000 lines of standard-library Python plus one HTML file. The ledger is deliberately a simple linear chain with periodic anchors, not a Merkle tree. The authors say the contribution is what gets recorded and cross-checked, not the cryptography.

How it works

flowchart LR
  A[Claude Code hook event] --> H[hook.py]
  S[SDK / HTTP agents] --> L
  H -->|PreToolUse| P{regex policy}
  P -->|deny: exit 2| A
  P -->|allow/flag| L[Hash-chained ledger]
  H -->|transcript lines| L
  L --> R[Rule flags]
  L --> J[LLM reviewer, no tools]
  J -->|verdict appended| L
  L -->|head hash| X[Off-machine anchor]

• Capture. hook.py runs on Claude Code lifecycle events: UserPromptSubmit, Pre/PostToolUse, PostToolUseFailure, SubagentStart/Stop, Stop and SessionStart/End.
• It reads new transcript lines exactly once, using a locked byte offset per transcript.
• Sub-agents are identified by agent_id/agent_type and by transcripts at <session>/subagents/agent-<id>.jsonl.
• Self-report often lands after the action it explains. Readers join the two by tool_use_id instead of reordering the ledger.
• Ledger. Each record holds seq, a timestamp, the event body and prev (the previous record's hash), all hashed over sorted-key canonical JSON.
• Credential-like strings are masked before hashing.
• Every append takes a file lock, recovers the last valid record by reading backwards (so a torn write survives a crash) and fsyncs.
• Gate. Regex deny/flag rules are matched against the Bash command or the file path. Writes outside the working directory are flagged.
• Cross-checks.
• Rule flags include unmentioned-file edits: writes to a file named in neither the prompt nor any earlier self-report.
• judge.py sends a compacted transcript to a separate model instance with no tools and gets structured JSON back.

Evidence

• E1, tampering (N=300 records, 200 trials per mutation class).
• The chain alone detects 100% of edits, deletes, swaps, forged inserts and torn writes.
• It detects 0% of tail truncation or full re-chaining. Anchors catch both.
• Detection of re-chaining follows P = 1 − (k−1)/2N, falling to 0.47 at an anchor interval of k=300.
• E2, overhead (2-vCPU container).
• Median hook cost is 23.9 ms on an empty ledger and 24.6 ms at 100k records, against a 10.3 ms bare Python start.
• Appends run at 2,436 records/s and verification at 35,099 records/s.
• Throughput stays around 2,000 records/s with 16 concurrent writers, and all chains verify.
• E3, gate. 18/44 harmful calls blocked and 3/40 benign calls wrongly blocked. Per category:

• E4, real runs. In 14 headless Claude Code 2.1 runs, the agent ignored all 4 planted injections and disclosed each one.
• E5, seeded-fault splicing. Over 63 traces:
• Rule flags caught 40/49 faulted traces.
• The LLM reviewer caught 98/98 reviewer calls on faulted traces and raised 1 false positive in 14 unmodified traces.

How to use it

The page names the components (hook.py, view.py, judge.py, observer.py) and says hooks are installed through a Claude Code settings file. It gives no install commands and no licence. The system, the evaluation harness (eval/) and all traces are said to be released.

Operational knobs:

• Fail-open by default. On an internal error the hook exits 0, letting the call proceed. Setting TRACEKIT_FAIL_CLOSED=1 denies calls instead, including when the policy file is unreadable.
• Bearer token. Required on every endpoint when the observer is exposed beyond localhost.
• Anchoring is on you. Without off-machine anchors, truncation and re-chaining are undetectable. Choose k from the formula in E1.

Limits

• The gate is a tripwire, not a sandbox. Trivial rewrites evade it, and it blocked 0/6 exfiltration attempts.
• No reasoning text in practice. The provider withheld all 27 thinking blocks, so self-report was limited to visible prose.
• The harness must be trusted to fire hooks faithfully.
• Side effects below tool-call level are invisible, for example a script that silently opens a socket. The authors defer that to OS-level enforcement.
• The evaluations are small: 14 real runs, 63 traces. Faults are spliced into real traces rather than produced by a misbehaving agent, and the reviewer is the same CLI and model family as the agent.
• Capture is evaluated only for Claude Code. Other agents connect through the SDK/HTTP API, but this was shown only as a demo.