Putting models to the secure coding test: Security rules

What it is

A single-run experiment by Datadog Security Labs, the second in a series, testing whether Project CodeGuard v1.4.0 security rules change the security of code written by coding agents. Three harness and model pairs each built the same "document portal" from one prompt: Claude Code with Sonnet 5, Cursor with Composer 2.5, and Codex with GPT 5.5. The prompt asked for a JavaScript and PostgreSQL app with JWT plus session cookies, roles, password reset, search, file upload, comments and profile editing. It did not say who may see uploaded documents.

How it works

CodeGuard was installed once per harness and then ran in the background. For Claude Code it came from the plugin marketplace, scoped to the project folder. For Cursor and Codex it came from a zip and the installation guide's commands. Supply Chain Firewall blocked outdated or vulnerable dependencies. Output was checked with Datadog Code Security (SCA, SAST, IaC) and a Sonnet 5 code_review pass. The no-rules baselines were reused from part one.

Results with rules

Composer 2.5's requireEnv helper applies the fallback before the production check, so the check never fires. Rules added rate limiting on its auth routes, which the baseline lacked.

GPT 5.5 also gained double-submit CSRF with Origin and Sec-Fetch-Site checks and magic-byte upload validation, fixing the baseline's CSRF and MIME problems. It pinned exact dependency versions, while the other two kept ^ ranges.

Sonnet 5 with rules showed a regression: the baseline already had revocation on logout, and the rules build moved the check to CSRF-protected routes only.

What to do

The author's recommendation is to put persistent guidance in the security rules or the global agent file, and to state access constraints in the prompt, for example "filter every document query by owner". The suggested global rule:

# ~/.config/agents/AGENTS.md
Always implement a dependency cooldown of 7 days unless otherwise specified. If the package manager does not support this, do not configure the dependency to automatically upgrade to the next version.

Caveats

Each model ran once, and the author notes that nondeterminism means other runs could differ. Findings come from automated scanning and one model's review, with full detail in the linked repository. The author concludes that the rules help but are not specific enough to guarantee a fix, and that the missing access requirement in the prompt explains the IDOR.