Skip to content

Back to the catalog

The Illusion of Secure LLM Code: Closing the Security Gap via Iterative Reprompting
arxiv.orgpaper

The Illusion of Secure LLM Code: Closing the Security Gap via Iterative Reprompting

This paper evaluates authentication code generated by five prominent AI coding assistants, combining static code analysis and dynamic penetration testing mapped against NIST SP 800-63B guidelines. It tests four prompting strategies (Basic, Secure, NIST-Based, and iterative Reprompting) and finds that basic and generically "secure" prompts consistently miss critical protections like brute-force resistance, session management, and password handling, while even a single-shot NIST-context prompt is structurally inadequate — only an iterative self-auditing reprompting loop achieves defense-in-depth. The authors argue enterprises must shift from single-shot prompt engineering to continuous, standards-driven verification pipelines for AI-generated auth code.

TL;DR Study of 5 AI coding assistants shows LLM-generated authentication code is insecure by default against NIST SP 800-63B unless forced through iterative self-auditing reprompting.
Open original ↗
Document

What it is

An empirical study (arXiv:2607.23710, Singh et al., Thapar Institute / Waterloo) that generates authentication code from five specific AI assistants and tests it against NIST SP 800-63B and OWASP. The models tested are named: Google Antigravity (Gemini 3.1 Pro), Cursor IDE (free tier), OpenAI Codex (GPT 5.3), and GitHub Copilot running Claude Sonnet 4.5 vs 4.6. The summary states the conclusion; this document is worth reading for the reproducible harness and the exact controls that failed.

How it works (the reproducible part)

The test constrains every model to build a Flask app on SQLite3 using raw SQL queries, no ORM, across four fixed views (landing, registration, login, dashboard). This is deliberate: stripping the ORM removes framework guardrails so you can see whether the model itself implements parameterization, sanitization, and session logic. Each prompt runs in a clean isolated chat session (functional requirements restated every time) to prevent context/memory bias.

Four prompt tiers escalate guidance: BP (novice, no security words), SP (adds only "secure and clean"), NP (SP plus NIST directives extracted from the SP 800-63B PDF via Google NotebookLM — the paper is explicit this is document-assisted extraction, not true RAG), and RP (re-supply the NIST guidance and force the model to list completed / partially completed / incomplete parameters, then finalize).

Evaluation is bi-modal: manual static review (KDF vs weak hash, salts, cookie flags) cross-referenced against a matrix, plus dynamic pen-testing on locally hosted builds using Burp Suite and HackingBuddyGPT for brute force, session hijacking, XSS/SQLi, and missing HTTP security headers.

Concrete NIST controls the study checks against

Useful as an audit checklist for your own AI-generated auth code:

  • Passwords: min 8 chars with MFA / 15 alone; must accept ≥64 chars; no forced complexity; no forced expiration; no hints/KBA; blocklist compromised passwords; allow paste; store with Argon2 or PBKDF2 + work factor.
  • Sessions: random session IDs ≥64 bits entropy; cookies tagged Secure, HttpOnly, SameSite (Lax/Strict); inactivity + absolute timeouts; server- and client-side secret erasure on logout.
  • Rate limiting: throttle failed logins (e.g. max 100 consecutive fails).
  • AAL reauth windows: AAL2 = 24h / 1h inactivity; AAL3 = 12h / 15min.

What to do

Treat the RP loop as the operational takeaway: single-shot prompting — even with NIST context pasted in — is structurally insufficient, so build a self-audit reprompt step ("list completed/partial/incomplete, then fix") into any pipeline emitting auth code. The controls above are the parameters that most consistently came back missing (brute-force resistance, session management, password handling).

Caveats

Static analysis was manual, not tool-generated, and dynamic results were recorded only for the relevant model-prompt cells — this is a source-inspection study against a hand-built matrix, not an automated benchmark. Model versions (Gemini 3.1 Pro, GPT 5.3, Sonnet 4.5/4.6) are a snapshot; findings are tied to those releases and the raw-SQL stress condition.

Organisations named here

  • NIST SP 800-63B

Technologies named here

  • Static code analysis
  • AI coding assistants
  • Dynamic penetration testing
  • LLM

Techniques named here

  • Basic prompting
  • Secure prompting
  • NIST-Based prompting
  • Reprompting

Related in AI Security