Let Them Steal: Trapping Large Language Model Extraction Attacks with Knowledge Honeypot

What it is

"Knowledge Trap" (Dai & Dong, Florida State University; arXiv:2606.15810v3, 29 Aug 2026) is a deception defense against model-extraction attacks (MEAs) on black-box LLM APIs. Code: github.com/LabRAI/knowledge-honeypot. The summary covers the concept and the headline 6.2% surrogate-agreement reduction; what follows is the mechanism, its construction parameters, and — most important for an engineer — the conditions under which it fails.

How it works

Three stages, all operating at the API level (query text, length, timestamps, session/API-key identity — no access to attacker logits, gradients, or intent):

• Detection. A per-query suspicion score combines a length signal (long structured queries), a keyword signal firing on extraction terms like define, explain, which, and a temporal signal that accumulates with sustained querying within a session. When the score exceeds threshold τ, breadcrumb injection turns on.
• Breadcrumb injection. The defender appends a short phrase naming a Honeypot Knowledge Graph (HKG) node, chosen as the unvisited node of highest out-degree. To resist template filtering, phrases are sampled from a pool of N=20 LLM-generated paraphrase templates.
• Self-reinforcing entrapment. Once the attacker queries a honeypot node, the response is served from the HKG and seeds further breadcrumbs. Because these concepts are novel to the surrogate, its uncertainty-based acquisition function rates them highly, so query allocation drifts monotonically into the HKG.

HKG construction (per domain): 1,500 nodes, 4,500 edges, minimum out-degree 3, 100% internal edges (no path back to task-critical knowledge). Candidates generated with gemini-flash-2.5 (temperature 0.3) across three categories — historical minutiae, peripheral taxonomy, niche procedural records — then keyword-filtered against benchmark terms and transferability-filtered with a probe model retaining only concepts whose downstream utility change is ≤ δ=0.02. Measured utility comes out at 0.000.

Preconditions / Caveats

This is the part the summary omits, and it bounds when the defense is worth deploying:

• Only response-adaptive attackers are trapped. The entrapment loop and the paper's Theorem 1 bound both rest on the attacker generating next queries from entities in prior responses. Section 4.8 acknowledges the effect "scales down" for attacker families that do not condition on response content — an attacker using a fixed query set is largely unaffected.
• Session-splitting evades detection. The temporal signal accumulates per session, so an attacker who spreads a fixed budget across independent sessions can delay or avoid activation (quantified in RQ5 / Appendix B).
• Domains evaluated. Table 2 reports only medical and financial HKGs (1,500 nodes each); the abstract also claims legal, but the statistics table does not include it.
• Defense is deliberately weak by design: the defender never blocks, rate-limits, or refuses queries — redirection only — which is why benign utility is preserved but also why a non-adaptive attacker faces no barrier.

Bottom line for an engineer: this is a research prototype, not a deployable control. It meaningfully raises cost only against uncertainty-driven, response-adaptive active-learning extractors that keep a single session, and the 6.2% agreement reduction is modest.