Putting models to the secure coding test: Plan vs default mode

What it is

Datadog Security Labs ran one prompt through three models, each in default and plan mode, and audited the six resulting apps. The models were Sonnet 5 (in Claude), Composer 2.5 (in Cursor) and GPT 5.5 (in Codex). The prompt asked for a JavaScript "document portal" with JWT plus session cookies, PostgreSQL, roles, password reset, search, file upload, comments, profile editing and a GitHub Pages workflow. It did not say that documents should be visible only to their owner. Audits used Datadog Code Security and Claude's code_review skill (Sonnet 5, top eight findings). The post is the first in a series, and the author's conclusion is that plan mode shows no meaningful link to more secure code.

Findings per build

• Sonnet 5, default: Findings were mostly functional bugs, with no authentication bypass. They included CR/LF in filenames breaking the Content-Disposition header and an admin self-protection check that does not guard the whole admin pool. The backend had CSRF double-submit, four rate limiters and revocable refresh tokens.
• Sonnet 5, plan: It added a hardening pass seen in no other run, with Prisma, rotating refresh tokens with reuse detection and strict environment validation. requireCsrf covered only session endpoints. sameSite 'strict' breaks the GitHub Pages plus separate API topology. Downloads were broken (a plain <a href> against a Bearer-protected route). requireAuth trusted JWT claims without a database check.
• Composer 2.5, default: It had MIME-only upload validation (no magic-byte check) and unescaped filenames in Content-Disposition. There was no JWT revocation, and the limit parameter was unbounded.
• Composer 2.5, plan: This was the worst build. jwtSecret falls back to the literal dev-secret-change-me with no startup check, so an unset JWT_SECRET allows token forgery and admin impersonation. Email changes need no password re-verification. Tokens are not rechecked against the database, and sameSite 'lax' conflicts with the cross-origin setup. The author says only this build had findings that could lead to complete compromise.
• GPT 5.5, default: When NODE_ENV is not 'production', password-reset tokens are returned in the JSON response and cookie sameSite/secure flags are silently downgraded. The default is 'development'. JWTs carry no password-version claim. The author rated this the worst of the default-mode builds.
• GPT 5.5, plan: It fixed the reset-token and cookie issues but added a TOCTOU race in first-admin provisioning (unlocked COUNT() then INSERT), a login timing side channel (bcrypt skipped for unknown accounts), no rate limiting on /api/auth/, and old reset tokens that stay valid after a new one is issued. The author ranked it second best of the six overall.

Cross-cutting results

IDOR was present in all six builds. GET, download and comment routes checked that a document exists but not who owns it, while DELETE usually did check. The author's reading is that a plan cannot supply a constraint the spec omits, and the plan-mode sessions asked about ORM and frontend choices but never about document visibility.

All six used parameterized queries. Only the two plan-mode builds that used Prisma (Sonnet 5 and Composer 2.5) get that protection from the framework. The other four rely on developers using placeholders every time. Every build used ^ version ranges and none used dependency cooldowns. Only the two Sonnet 5 builds chose current, non-vulnerable dependencies.

What to do

The author ran every install through Supply Chain Firewall, an open source tool that blocks malicious or vulnerable npm and Python packages:

scfw run npm install

It blocked multer@1.4.5-lts.2 (eight high vulnerabilities), nodemailer (at least one high), vite@5.4.21 (one high, two medium) and esbuild@0.21.5 (one medium, transitive via vite). The agents then moved to multer 2.2.0 and nodemailer 9.0.1. Remediations the audits named:

• Centralize ownership checks in shared middleware, not per-controller helpers.
• Fail startup when JWT_SECRET is missing or equals the placeholder.
• Decouple debug-mode token exposure from ambient environment variables.

The author says the prompt mattered more than the mode, so state authorization rules explicitly, for example "documents must be only visible to their owner".

Caveats

Each configuration was run once, so the differences between builds are single samples. The author notes that other runs may have had a hardening step that was not visible. The next post in the series will test security-specific skills and prompts. Full code and a detailed findings breakdown are in the author's repository, which this summary does not reproduce.