Applying Security by Design at the Point of Execution: How Governed Security Requirements Affect the Security of AI-Generated Code

What it is

A single-author empirical study (Farinha, Shiftleft) of delivering security requirements to a code-generating model at generation time. Requirements come from the SbD-ToE manual's knowledge graph (release 0.10.1) and are served by an MCP server. The agent passes the task text, a mode (code generation) and a risk level. The selector matches terms to requirement families and to the ten "slices" of the AppSec Core ontology. It returns control objectives, base requirements, controls and evidence expectations, each with a stable ID. The payload contains no code; the author checked this in the DualGauge audit logs, where none of the 72 results contained source.

What the numbers add

The summary gives the headline rates. Details it omits:

• DualGauge (59 tasks, Claude Code + claude-opus-4-7, judged by gpt-4o): secure@1 gained 23 tasks and lost 3 (+33.9 pp, CI +18.6 to +48.1). The gain is concentrated: the per-task median change is 0 and the mean is +18.8 pp, so about half the tasks drive the pooled result. Biggest movers: path traversal 16→30 of 34 tests, log injection 0→9 of 9, XSS 14→19 of 19.
• Functional cost on DualGauge: pass@1 fell from 25 to 20 tasks (4 improved, 9 worse), and the pre-set 5 pp non-inferiority margin was not met. The joint metric went 12→18 (p = 0.109, inconclusive).
• BaxBench has three runs, not one. "Secure among correct" with MCP versus baseline: direct call (FastAPI) 65%→86%, agent FastAPI 70%→83%, agent Flask 75%→84%.
• MCP matched the Oracle prompt only in two of three runs. The agent FastAPI oracle reached 91%, and Flask MCP was 84% against 83%.
• The denominator shrinks. Functionally correct solutions with MCP dropped from 26→21 (direct), 23→18 (agent FastAPI) and 24→19 (Flask). Correct-and-secure counts barely moved: 17→18, 16→15, 18→16. "Secure among correct" is descriptive and unpaired, and the gain mostly comes from fewer correct solutions rather than more secure ones.
• Unassessed cells grew. Cells where security tests didn't complete rose from 5 to 10, 6 to 11, and 5 to 10 under MCP.
• Paired BaxBench test: 4 cells were insecure only without requirements and none only with them (p = 0.125, not significant).
• Cost: BaxBench runs cost $14.88 (FastAPI agent), $14.49 (Flask agent) and $32.97 (direct call).

Preconditions

• The risk level was fixed at L2 for every task. Other levels were not tested.
• Direct-call BaxBench used a separate Claude Sonnet 5.5 agent to retrieve requirements and insert them into the reminder slot. It is not a pure single-prompt setup.
• The code generators were claude-opus-4-7 (agents) and Claude Opus 4.5 (direct call), with one generation per task.

Caveats

• Only the end-to-end effect was measured; the selector, the knowledge-base content and the delivery mechanism were not separated.
• The selector's completeness and correctness were not checked, and the knowledge base's content has not been reviewed by domain experts.
• The registration is internal and self-held. The McNemar tests and most intervals were added post hoc without multiple-testing correction.
• An executor defect was repaired after generation and the grounded condition re-run. The security conclusion holds either way, but the joint metric depends on the repair.
• The stricter code fails functional tests that fix values the task never states. The author attributes this case by case in a post-hoc analysis.

Should you act?

Treat it as evidence that giving the model the applicable requirements helps on security tests, not as proof of a free win. If you try this, track functional pass rate alongside security, since both benchmarks showed more security failures avoided at the price of fewer functionally correct outputs.