APort Vault: Benchmarking AI Agent Payment Authorization with the Open Agent Passport

What it is

APort Vault (arXiv 2609.22076v1, submitted 18 Sep 2026) benchmarks payment-authorization safety by replaying 4,371 human-written CTF attacks against a live tool-using payment agent, with and without a deterministic pre-action check implementing the Open Agent Passport (OAP) spec. The existing summary has the headline (140/76,842 unauthorized transfers model-alone vs 0/69,297 behind the layer; 25,370 legitimate payments allowed). What the summary omits is the internal structure that tells you how much to trust that zero — and a confound the authors flag themselves.

What the numbers actually rest on

The clean comparison is the matched triple: on 68,970 identical (model, prompt, track) triples, unauthorized transfers were 105 with the model alone vs 0 behind the layer. The full 140-vs-0 figure spans unmatched cohorts; the 105-vs-0 is the apples-to-apples version.

The zero spans 790 source sessions, giving a per-session upper bound of 0.38% — i.e., the layer's failure rate isn't demonstrably zero, only bounded below that. Worth stating plainly when someone reads "0/69,297" as absolute.

It was not achieved by refusing everything: behind the layer the policy denied 187 of 25,640 transfer calls it evaluated, 148 of them for a forbidden recipient, and let 25,370 payments through.

Request-rate structure

Whether the agent even attempts a transfer is driven far more by policy configuration than by model choice. Model-alone request rates: 10.9% at Level 1, 3.0% at Level 2, 0.1% at Level 3, 79.4% at Level 4. On the 1,293 Level 4 prompts (each run on all 14 models), request rates cluster tightly, 71.2%–84.3%, and 809 prompts (62.6%) drew a request from all fourteen models, each ending in a successful payment to the level's allowlisted recipient. The takeaway: at a level designed to elicit payment, models behave nearly identically — differentiation lives in the policy layer, not the model.

Caveats

• The authors state the key confound directly: each attack exists at exactly one configuration, so policy and attack cohort vary together. You cannot cleanly separate "Level 3 is safer" from "Level 3 drew easier attacks." The matched-triple analysis is their answer to this, but the cross-level rates above inherit the confound.
• They deliberately report five distinct events per evaluation rather than one collapsed score, arguing a single agent-safety number "does not survive review" — relevant if you plan to cite or reproduce this.
• Full release: the 225,964 evaluations, level passports, scoring code, and analysis script.