Applying Security by Design at the Point of Execution: How Governed Security Requirements Affect the Security of AI-Generated Code

What it claims

Supplying requirements at generation time closes much of the gap that secure-code benchmarks attribute to the model. The paper splits that gap into two kinds of failure:

• Capability failure: the model cannot implement a security property.
• Specification failure: the model did not guess that the property was required.

The requirements come from the SbD-ToE knowledge graph, release 0.10.1, served over MCP. The paper's motivating evidence is BaxBench's own leaderboard, where naming the tested weaknesses raises Claude Opus 4.5's correct-and-secure rate from 0.561 to 0.686 with the same model.

How it works

The MCP server exposes a selector tool with three inputs:

• task text
• mode (here, code generation)
• risk level (L1, L2 or L3)

It matches terms in the task against requirement families and the ten "slices" of the AppSec Core ontology, such as input validation. It returns an activated scope: control objectives, base requirements, controls and evidence expectations. Each item carries a stable ID and a reference to its source in the manual. The payload contains no code; in the DualGauge audit logs, none of the 72 server results contained source code. Risk level was fixed at L2 for every task, because the benchmarks do not classify their tasks.

There are two delivery modes:

flowchart LR
  T[Task text] --> S[MCP selector, L2]
  S -->|agent mode| A[Claude Code agent writes code]
  S -->|direct-call mode| R[Sonnet 5.5 agent picks requirements]
  R --> P[Inserted in BaxBench reminder slot]
  P --> O[Single Opus 4.5 API call]

• Agent mode: the coding agent calls the tool itself. This was used for DualGauge and the BaxBench agent runs, with Claude Code and claude-opus-4-7.
• Direct-call mode: a separate agent (Claude Code with Claude Sonnet 5.5) retrieves the requirements. BaxBench's unchanged code path then makes one Opus 4.5 call with extended thinking.

Evidence

DualGauge (59 tasks, gpt-4o judge, baseline is the same agent without MCP):

The security gains were largest in three weakness types:

• path traversal: 16 → 30 of 34 tests
• log injection: 0 → 9 of 9
• XSS: 14 → 19 of 19

The per-task median change in security tests passed is 0, so roughly half the tasks carry the gain.

BaxBench (28 scenarios, executed exploits). The 65% → 86% headline is the direct-call FastAPI run, where functionally correct solutions fell from 26 to 21. The other two runs:

Insecure-among-assessed dropped to 17–18% in every MCP run.

Paired comparison: 0 cells became insecure only with MCP, against 4 that were insecure only without it (p = 0.125, not significant). Correct-and-secure counts barely moved: 17→18, 16→15 and 18→16.

Cost: $14.88 for the FastAPI agent run, $14.49 for the Flask agent run, and $32.97 for the direct-call run.

How to use it

The page names the components but gives no install command or endpoint:

• the SbD-ToE manual and knowledge graph, described as the author's open-source work
• the MCP server
• the selector tool taking task, mode and risk level

The DualGauge condition also used "skills and an activation instruction" alongside the server. The effect is estimated end-to-end, so no part is measured separately.

Limits

• Functional regressions. Stricter code fails functional tests that pin values the task never states. The attribution of these failures is post hoc, case by case.
• Fewer cells assessed. Not-assessed cells roughly double under MCP on BaxBench (5→10, 6→11, 5→10). This complicates the security comparison.
• Small samples. There is one generation per task (k=1) and one model per experiment. The models differ across experiments: Opus 4.7 for the agent runs, Opus 4.5 for direct call.
• Analysis choices after the run. The McNemar tests and most confidence intervals were added after the run, without correction for multiple testing. A DualGauge executor defect was repaired after generation, and the joint metric depends on that repair.
• Self-registration. Protocols were registered in the author's own internal commits.
• Author's own knowledge base. The author also wrote the knowledge base. Its content has not been expert-reviewed. Only its structure was validated, with OWL and SHACL.
• Selector not validated. The selector's completeness and correctness are not checked.
• Other risk levels untested. L1 and L3 would return different requirements and were not tested.