llm-security (greshake)

What it is

The proof-of-concept repository backing Greshake et al.'s ArXiv paper on indirect prompt injection — injections delivered not by the user but through content an LLM-integrated app retrieves (web pages, documents, emails, code snippets). The demos span GPT-4 (Bing and synthetic apps) via ChatML, GPT-3/LangChain apps, and code-completion engines like Copilot. Its central claim: prompt injection can be as powerful as arbitrary code execution, and retrieval turns "updating facts" into executing untrusted adversary-authored code on the model with full privileges.

How it works

The attacker plants instructions in content the model will later ingest. The repo demonstrates several distinct techniques beyond a single hijack:

• Hidden delivery — the injection sits in a side channel (e.g. a Markdown/HTML comment on a Wikipedia page) invisible to a human viewing the site but read by the model.
• Multi-stage payloads — a tiny injection in bulk content makes the LLM autonomously and invisibly fetch a larger payload.
• Remote control / C2 — a compromised agent retrieves fresh instructions from an attacker's command-and-control server (via search keywords or direct URL), enabling a bidirectional backdoor.
• Wormable spread — an agent that reads/composes email and can access contacts propagates the injection to other LLMs reading inbound messages.
• Cross-session persistence — payload stored in the agent's key-value "memory"; recalling the last conversation re-poisons it.
• Code-completion targeting — injection placed in a comment in a package the developer opens; the completion engine pulls it into context and can be biased to suggest malicious or subtly vulnerable code.

flowchart TD
  A[Untrusted retrieved content] --> B{Injection present?}
  B -->|no| C[Normal answer]
  B -->|yes| D[LLM executes embedded instructions]
  D --> E[Fetch larger payload]
  D --> F[Exfiltrate/alter user data]
  D --> G[C2 remote control]
  D --> H[Write to memory: persists across sessions]
  D --> I[Send email: spreads to other LLMs]

Preconditions / Affected

The model must have some pathway that ingests external content — retrieval/search, plugins, email reading, or a code-completion context window that scoops snippets from open files. Persistence needs a memory store; worming needs send/contacts capabilities. The authors expect the vectors to also apply to ChatGPT plugins and other application-integrated LLMs.

Caveats

This is a proof-of-concept repository, not a mitigation guide — it proposes no fixes, only demonstrates impact. The authors frame spread/worming and robust context-window persistence as directions attackers could pursue and explicitly call for further investigation of real-world generalizability. Treat it as a threat-model and red-team reference rather than a remediation source.