Putting models to the secure coding test: Security rules

What it claims

Datadog Security Labs ran Project CodeGuard v1.4.0 rules against three coding agents building the same document portal. This is the second post in a series; the first compared plan mode with default mode. Only GPT 5.5 (Codex) produced a meaningfully more secure app with the rules. The author's explanation is that rules do not supply missing requirements. The prompt never said who may read uploaded documents, and "build this securely" does not become "filter every document query by owner" unless someone writes that down.

How it works

• Same prompt for every run. It was reused from the first experiment: JavaScript/npm, JWT plus session cookies, PostgreSQL, multiple roles, password reset, search, file upload, comments, profile editing, admin user management, and a GitHub Pages deploy workflow.
• Rule installation. CodeGuard came from the plugin marketplace for Claude Code, scoped to the project folder. Cursor and Codex used the zip-file install from the CodeGuard guide.
• Package installs. Supply Chain Firewall (SCFW) blocked malicious and vulnerable packages during generation.
• Scanning and review. Datadog Code Security ran SCA, SAST and IaC scans. Claude's code_review skill, running on Sonnet 5, did the deeper audit of every build.
• Baseline. The no-rules results were reused from the first post.

Evidence

Each model ran once.

Sonnet 5 (Claude Code). The rules produced no meaningful improvement and one regression.

• The plugin added a plan-approval step and a self-verification step. No other harness did this.
• All queries are parameterized, and React escapes rendered output.
• The cross-user document IDOR remained.
• New regression: logout and deactivation are checked only in the CSRF middleware, which runs only on mutating requests. A revoked user keeps read access for the 15-minute access-token lifetime.
• Session fingerprints (user agent, IP) are captured but never compared.
• The token refresh endpoint has no CSRF check.
• Other low-severity findings: rate limiting keyed only by IP, an unused COOKIE_SIGNING_SECRET, and a 409 account-enumeration oracle on registration.
• It started with outdated dependencies, which SCFW caught.

Composer 2.5 (Cursor). This was the weakest of the three.

• Critical: JWTs are checked only for signature and expiry. They are stored in localStorage and stay valid for up to 24 hours after logout or password reset.
• JWT_SECRET and SESSION_SECRET fall back to hardcoded values. The rules prompted a requireEnv production check, but the fallback is applied before the check, so it never fires.
• Uploads are validated by client-supplied MIME type and extension.
• The IDOR remained. The only substantive change from the default build was auth rate limiting.

GPT 5.5 (Codex).

• No IDOR.
• Fixes over its own default build:
• double-submit CSRF with Origin and Sec-Fetch-Site checks (the default build had a real CSRF bug in uploads),
• magic-byte upload validation,
• Argon2id password hashing,
• server-side session revocation,
• exact pinned dependency versions instead of ^ ranges.
• Remaining issues:
• new reset requests do not invalidate older unexpired reset tokens,
• Argon2id hashing runs before the reset token is validated,
• a computed ip_hash is never compared, so a stolen cookie works from any IP if the User-Agent is spoofed.
• A broad manager role and a single global auth rate limit were judged design choices, not vulnerabilities.

How to use it

The full code for each build and a detailed vulnerability breakdown are in a linked repository. The author recommends putting persistent rules in global agent config rather than in each prompt, and proposes this addition:

# ~/.config/agents/AGENTS.md
Always implement a dependency cooldown of 7 days unless otherwise specified. If the package manager does not support this, do not configure the dependency to automatically upgrade to the next version. 

Limits

• Each configuration ran once. The author notes that output is nondeterministic, so other runs may differ.
• The auditor is Sonnet 5, which is also one of the models under test.
• The prompt deliberately leaves document access unspecified, so the IDOR result partly measures how each model handles an ambiguous requirement.
• SCFW changed the dependency outcomes in every run.
• Only CodeGuard v1.4.0 was tested. No other rule sets, and no prompts with explicit access requirements, were tried.