Do Defenses Against LLM Extraction Work Across Attacks? A Lifecycle Benchmark of Black-Box Model Extraction

What it is

A benchmark from Florida State University and collaborators (code at github.com/sliu11-byte/MEA-Bench) that evaluates black-box LLM model extraction attacks and defenses under matched conditions. It covers six extraction attacks (SeqKD, LoRD, SODA, GAD, Model Leeching, QEDKS), ten defenses, and two adaptive attacks (DIPPER paraphrasing, English–French–English back-translation).

How it works

The benchmark organizes extraction into four stages: query acquisition, victim interaction, surrogate training, evaluation. Attacks are grouped by whether they use fixed-pool query selection (SeqKD, LoRD, SODA, GAD) or method-specific query generation (Model Leeching via prompt templates; QEDKS via seed-query expansion), and by whether surrogate training uses plain response-supervised SFT (SeqKD, QEDKS, Model Leeching) or method-specific objectives (LoRD's pairwise optimization against sampled candidates, SODA's DPO on teacher–student preference pairs, GAD's joint discriminator training). Defenses are grouped by intervention point rather than a unified score: query detectors (MMD, PRADA, SEAT) classify traffic before victim inference; response-modification defenses either degrade supervision value (anti-distillation: ADS, DOGe, Trace Rewriting) or embed inheritable signals (provenance: ADFP, GINSEW, Radioactivity); post-training verification (DuFFin) performs lineage checks via diagnostic probes without touching responses. Adaptive attacks insert paraphrasing/back-translation between the protected response and surrogate training, then the paper jointly measures the resulting surrogate's provenance-detector score, capability, and fidelity — rather than scoring the rewritten text alone.

flowchart LR
    Q[Query acquisition] --> V[Victim interaction]
    V --> D{Defense applied?}
    D -->|no| S[Surrogate training]
    D -->|response modification| P[Protected response]
    P --> A{Adaptive rewrite?}
    A -->|no| S
    A -->|paraphrase/back-translate| S
    S --> E[Evaluation: capability, fidelity, detector score]

Preconditions / Affected

Applies to LLMs deployed via text-only APIs, where attackers have no access to parameters, gradients, hidden states, token probabilities, or private training data, and (when a defense is active) no knowledge of secret keys, watermark rules, or detector thresholds.

What to do

The paper's four research questions frame the actionable use of the benchmark: (RQ1) compare attack effectiveness under controlled budgets/configs; (RQ2) measure whether each defense meets its specific objective (query classification accuracy, surrogate degradation, provenance detection, lineage separation) and the utility cost imposed; (RQ3) test whether paraphrasing/back-translation can strip provenance signal while surrogate capability/fidelity survive; (RQ4) track how capability, fidelity, and output quality scale with query budget for SeqKD, SODA, Model Leeching, and QEDKS specifically. Teams evaluating an extraction defense should pair its security metric with surrogate capability/fidelity rather than trust a standalone defense score, per the paper's explicit design choice.

Caveats

The fetched text cuts off mid-methodology (defense evaluation section, query detection calibration) before presenting actual benchmark results/numbers — no concrete capability, fidelity, or detection scores are given in the retrieved content.