Putting models to the secure coding test: Plan vs default mode

What it claims

Datadog Security Labs built the same "document portal" app six times: Sonnet 5 (Claude), Composer 2.5 (Cursor) and GPT 5.5 (Codex), each in default mode and plan mode. Every build had an insecure direct object reference (IDOR): document view, download and comment routes checked that a document existed but not who owned it. Delete routes usually did check ownership. The author traces this to the prompt, which never said documents should be visible only to their owner. None of the plan-mode sessions asked about document-level access, although they did ask about ORM and frontend framework choices. The author's ranking: Composer 2.5 plan mode was the least secure, GPT 5.5 plan mode was second-best, and GPT 5.5 default mode was the worst of the three default-mode builds.

How it works

Each run started in an empty folder, with an instruction not to reference files outside it. The prompt asked for JavaScript (npm), JWT plus session cookies, PostgreSQL, login, multiple roles, password reset, search, file upload, comments, profile editing, and a GitHub Pages deploy workflow. In plan mode the author always picked the recommended or top option. Agents had to run scfw run npm install (Supply Chain Firewall) instead of npm install. Findings came from Datadog Code Security (SCA, SAST, IaC) and from Claude's code_review skill running on Sonnet 5 across all six builds. The code and a detailed vulnerability breakdown are in a linked repository.

Evidence

How plan mode changed each model's output, compared with its default-mode build:

• Sonnet 5. Plan mode added Prisma, Zod validation, rotating refresh tokens with reuse detection, and a security hardening pass not seen in other runs. It also had gaps. requireCsrf covered only session endpoints, not the document and user routes. sameSite: 'strict' breaks the cross-origin GitHub Pages plus API setup. Downloads did not work, because a plain <a href> cannot send a Bearer header. JWT claims were trusted without a database recheck. Neither Sonnet build had a full authentication bypass.
• Composer 2.5. Plan mode was worse than default. jwtSecret fell back to the literal 'dev-secret-change-me' with no startup check, which allows token forgery and admin impersonation. PATCH /api/users/me allowed email changes without re-entering the password. The default build had MIME-only upload validation, Content-Disposition header injection, no JWT revocation, and stored JWTs in localStorage.
• GPT 5.5. The default build returned live password-reset tokens in JSON and dropped the cookie sameSite/secure flags whenever NODE_ENV was not 'production'. Plan mode fixed both problems. It introduced a race condition in first-admin provisioning (an unlocked COUNT(*) then INSERT), a login timing side channel (bcrypt skipped for unknown users), no rate limiting on /api/auth/*, and no CSRF protection.

Supply Chain Firewall blocked multer@1.4.5-lts.2 (eight high vulnerabilities) in four runs. It also blocked vite@5.4.21 with its transitive esbuild@0.21.5, and a vulnerable nodemailer. The agents upgraded to multer 2.2.0 and nodemailer 9.0.1. All six builds used parameterized queries; only the two Prisma builds got this enforced by the framework. No build used dependency cooldowns, and all pinned versions with ^. Only the two Sonnet 5 builds started with up-to-date dependencies.

Limits

Each configuration was run once on a single prompt, and the results are not scored or counted. The reviewer was Sonnet 5, which is also one of the tested models, and its review is normally capped at eight findings. Always choosing the recommended plan option leaves other plan paths untested. The author says Supply Chain Firewall had a small effect on the security of the output. Each model ran in only one tool (Claude, Cursor or Codex), so the model and the tool are not separated. The author does not recommend a model. A follow-up post will test security-specific skills and prompts.