Deletion, Substitution, Repair and Transplant: Component Attribution in BARBICAN, a Layered Prompt-Injection Gateway for MCP

What it is

A paper by İpek Abasıkeleş-Turgut and Edip Gümüş (arXiv 2610.33198v1, 23 October 2026). It re-executes BARBICAN, a fail-closed MCP prompt-injection gateway from the authors' own group, on its 25,000-prompt benchmark. The aim is to test whether component ablation by deletion, the usual way layered defenses credit their stages, measures what a stage contributes. The authors say it does not, and propose a reporting standard.

How it works

The gateway routes inputs through rules, a semantic stage, a neural arbitration stage and a fail-closed default. The authors ran fifteen live re-executions on fixed data and hardware, then a second series of twelve. The control run reproduces the recorded confusion matrix (88.26% accuracy) exactly. Three operations were applied to the arbitration stage:

• Deletion: the band fails open and 673 detections are lost.
• Substitution: pinning the verdict to a constant moves only 12 to 140 of 25,000 decisions. Real influence is bounded at 152 decisions (0.61%). That is a 4.8-fold gap from deletion.
• Repair: the stage had spent the whole recorded evaluation reading tokenizer-mangled text full of unknown-token markers (AUC 0.392). Fixing it gives AUC 0.855, yet system recall falls from 80.67% to 80.34% (48 detections lost, 7 gained). A stub that always answers "malicious" scores 80.77%.

The paper explains this through the stage's input and the enforcement rules:

• Upstream routing leaves the arbitration band 96.82% malicious, against 30.11% outside it. A constant "malicious" verdict therefore gets 96.82% precision in the band.
• A "malicious" verdict ends in a block. A "benign" verdict must pass further conditions before releasing anything. With the gateway's own semantic scores those conditions passed 12 times in 7,453 when every verdict was forced benign, and never in the control.
• Deletion loses detections mostly because the fail-closed rule for unresolved reviews is swapped for a looser release rule.

A fourth operation, transplant, put a public classifier (AUC 0.916 on the band, against 0.368 for the gateway's own) into the arbitration stage. It changed 16 of 25,000 decisions. Put in the semantic stage instead, it changed 1,015, because there it decides what gets routed.

The paper traces the real detections to an undocumented fusion stage, hand-written policy branches keyed on formatting strings (four of five fire only on synthetic samples), a similarity memory whose added blocks are correct 55.6% of the time, and the fail-closed conversion itself. In the authors' earlier gateway, the review model's band is 92.94% malicious and a constant verdict is as accurate as the review model.

Corpus findings: one public collection is labeled by prompt form rather than harm, one has attack strings embedded in generated HTML, and one has an appended line of challenge metadata. Against four public detectors with operating points chosen off the test data, the gateway leads on its own distribution. Under distribution shift the strongest public detector beats it on detection and false alarms together.

What to do

The paper's reporting proposal, which it says costs one extra run each:

1. Report the class balance of the band routed to any stage that gets routed rather than complete traffic.
2. Report substitution (constant verdict) results next to deletion results.
3. Where possible, check a stage's quality in isolation, since none of the recorded accuracy, precision, recall, F1 or correlation figures showed the tokenizer fault.

Caveats

• The authors built the system and argue this access was needed for the interventions.
• It is not a ranking of complete defenses. No competing gateway was rebuilt under this protocol, and the four detectors are single-classifier baselines.
• LLM Guard and NeMo Guardrails were audited for what deletion credits them, not ranked.
• The size of the deletion-substitution gap depends on how weak-evidence releases are enforced (Section 5.4).
• The fetched text is truncated partway through Section 2.3, so results from Sections 5 to 8 are taken from the abstract and introduction only.