ORCAGen: Orchestrating Context-Aware Malware Deception with RAG-Guided Generative AI

What it is

ORCAGen (Ahmed, Sajid, Taylor, Araujo, Islam; Towson, IBM Research, UMBC; arXiv 2610.12415v1) is a framework for building per-malware deception playbooks for endpoints after a suspicious process has been identified. The idea is to keep a suspect process alive under controlled deception instead of killing it immediately, so defenders keep visibility into what it does. It is not a detector, remover or access-control mechanism.

How it works

Construction and enforcement are separated, and a validation gate sits between them.

flowchart LR
    KB[Two KBs: procedures + active defenses] --> RAG
    RAG --> L1[LLM1: PoC malware]
    RAG --> L2[LLM2: orchestration code]
    L1 --> V{Deception works against PoC?}
    L2 --> V
    V -- no --> R[Refine]
    R --> L2
    V -- yes --> P[Playbook / Super DLL]
    P --> E[Inject into suspect process, no LLM at runtime]

• Knowledge bases. There are two JSON KBs. Malware Procedures holds malware family, attack technique, behavioral description and the APIs or artifacts involved. Active Defenses holds the deception objective, hook or monitor, trigger condition, safety constraint and expected response.
• KB sourcing. Sources come from academic papers, threat reports, malware blogs and technical docs. Collection combines exact-phrase and multi-term lexical queries with semantic search via Perplexity AI, then manual filtering. Seeds include MITRE ATT&CK T1056, T1003 and T1486, and APIs such as GetAsyncKeyState.
• Generation. RAG injects retrieved entries into structured prompt templates (objective, output format, context, system prompt, example). No fine-tuning is involved. Separate LLM roles generate the PoC malware and the orchestration code.
• Validation. The orchestration code is compiled and run against the PoC. Only logic that passes goes into the playbook.
• Runtime. A validated "Super DLL" is injected into suspicious or policy-selected processes. Example interventions are false keystrokes, redirecting credential access to honey artifacts, decoy file names, and simulated success or failure on destructive file operations.

Preconditions / Affected

• The defender needs endpoint control to inject the DLL, monitor selected API and file-system events, and keep whitelists of trusted applications.
• An external signal must nominate candidate processes (EDR, sandbox verdict, analyst rule or policy).
• The target is user-space malware whose activity passes through userland APIs or artifacts: keyloggers, infostealers and ransomware.

Caveats

• Out of scope. Kernel-level malware, malware that never touches monitored user-space behavior, and anti-hooking tampering are outside the evaluation. Anti-hooking could be handled only by adding the relevant procedures and countermeasures to the KB.
• Coverage limit. The authors state that runtime enforcement covers only behaviors and interventions represented in the KB and validated beforehand.
• Validation is against generated PoCs. The deception is tested against LLM-written PoC malware, not the real sample. The paper separately reports results on 150 real samples.
• Model comparison. The abstract reports that GPT-5.5 needed the fewest refinements with no observed hallucinated APIs, and that Gemini 3.5 Flash had the lowest response time and runtime overhead. The fetched text cuts off before the evaluation section, so I cannot give the numbers, the deception-effectiveness results, or the Claude Sonnet 4.5 and Qwen3-Coder figures. Read Section V before relying on them.
• Not stated. The page does not say whether the KB or code is released. It points to a reference [19] for the full schema and entries.