Deletion, Substitution, Repair and Transplant: Component Attribution in BARBICAN, a Layered Prompt-Injection Gateway for MCP

What it claims

BARBICAN is a fail-closed MCP gateway built by the authors' own group at Iskenderun Technical University. Its neural arbitration stage barely affects final decisions, and a deletion ablation overstates its worth by about 4.8 times compared with substitution. Deleting the stage costs 673 detections out of 25,000. Pinning its verdict to a constant, with routing and enforcement unchanged, moves only 12 to 140 decisions. The two sets do not overlap, so the stage's real influence is bounded at 152 decisions (0.61%). A three-line stub that always answers "malicious" reaches 80.77% recall, higher than both the broken classifier (80.67%) and the repaired one (80.34%). Repair cost 41 detections net: 48 lost and 7 gained. On the four public corpora inside the benchmark, repair moved recall by 0.00 points.

How it works

The gateway is a cascade: deterministic rules, a semantic stage, a neural arbitrator, and a fail-closed default. The arbitrator only sees what earlier stages could neither clear nor condemn. That band is 96.82% malicious, against 30.11% outside it. Enforcement is asymmetric. A "malicious" verdict always ends in a block. A "benign" verdict releases an input only through one narrow branch gated on semantic scores. That branch never fired in the control run and fired 12 times in 7,453 when every verdict was forced benign.

flowchart TD
  A[Band input] --> B{Arbitrator verdict}
  B -->|malicious| X[Block]
  B -->|unresolved| X
  B -->|benign| C{Semantic-score conditions}
  C -->|almost never pass| R[Release]
  C -->|fail| X
  D[Stage deleted] --> E[Band fails open: looser allow rule]

Deletion takes the band out of review, so the fail-closed rule stops applying and a looser release rule takes over. Most of the 673 lost detections come from that rule change, not from the classifier.

The authors traced where the reported detections actually come from. Sources include an undocumented fusion stage and hand-written policy branches keyed on formatting strings. Four of those five branches fire only on synthetic samples, never on public corpora. A similarity memory adds blocks that are correct 55.6% of the time, and the fail-closed conversion also contributes. None of these appear in the system's own architecture description.

Evidence

• Baseline: 88.26% accuracy on a balanced 25,000-prompt benchmark. A control run reproduces the recorded confusion matrix sample for sample. Re-execution on different hardware exchanged 2 of 25,000 decisions.
• Fifteen live runs covered deletion, substitution and tokenizer repair. A second series of twelve runs repeated the control three times, isolated a post-allow escalation step and swept the semantic threshold.
• Transplant: an off-the-shelf public classifier with AUC 0.916 on the band (the gateway's own classifier scores 0.368 on the same band) changes 16 of 25,000 decisions in the arbitration stage. It labels 202 benign band inputs benign, and none is released. The four inputs it does release are attacks. Placed in the semantic stage instead, the same classifier changes 1,015 decisions, because there it decides routing.
• Four public detectors were compared at operating points chosen away from test data. BARBICAN leads on its own benchmark distribution. Under distribution shift, the strongest detector beats it on both detection and false alarms.
• Corpus audit: one public collection is labelled by prompt form rather than harm. A second has attack strings embedded in generated HTML that the original benchmark lacks. A third carries an appended line of challenge metadata.
• The authors' earlier three-layer gateway has a review band that is 92.94% malicious, and a constant verdict there is as accurate as the review model.
• LLM Guard and NeMo Guardrails were audited component by component and as full stacks, to see what deletion credits in non-routing stacks.

How to use it

The authors propose two reporting rules for layered defenses. Report the class balance of the band each routed stage receives, because it caps the stage's possible contribution. Report substitution results beside deletion results. Each costs one extra run. The page does not name the public detectors or give code or artifact locations.

Limits

This is not a ranking of complete defenses. No competing gateway was rebuilt under this protocol, and the four external detectors are single classifiers. The authors built BARBICAN themselves; they present this as a requirement for access to source, configuration history and audit trail. The fetched text stops partway through related work, so Section 5 tables, the second-platform results and the Section 6 diagnosis of why no recorded metric exposed the faults were not available to check.