DITTO: A Context-aware Pickle-based Pre-Trained Model Scanner for Effective Security Audits

What it is

DITTO is a pre-load scanner for Pickle-based pre-trained models (PTMs). It emulates the Pickle VM (PVM) instead of matching on imported global names, and it also introduces an attack class that existing scanners miss: extension registry exploitation. The paper also provides PickleBench (959 benign and 92 malicious models) for comparing scanners.

How it works

Two components:

• Trace generator. It follows the same loading path the ML framework uses to find the embedded Pickle stream(s), including ZIP, GZ and TAR packaging. It then emulates the stack-based PVM and records each state transition without executing security-sensitive operations.
• Intention analyzer. It pulls the security-critical states out of the trace and runs semantic analysis over how each one is used, to judge whether it is legitimate. The paper says different semantic analyzers change classification performance. The extracted states and the reasoning report serve as audit evidence.

The paper groups the sensitive PVM operations into three types:

• Import: GLOBAL, STACK_GLOBAL, EXT1, EXT2, EXT4
• Invoke: INST, OBJ, NEWOBJ, NEWOBJ_EX, REDUCE, BUILD
• Alter: APPEND(S), SETITEM(S), ADDITEMS

The registry blind spot. EXT* opcodes resolve a global through copyreg registries, so the target depends on earlier state changes. The paper's example imports copyreg._inverted_registry, uses SETITEM to map index 1 to ("os", "system"), and then an EXT opcode resolves to os.system. The dangerous name never appears as a plain import. A scanner that only reads the identities of imported globals cannot see it.

flowchart LR
    A[GLOBAL copyreg._inverted_registry] --> B[SETITEM: 1 -> os.system]
    B --> C[EXT1 index 1]
    C --> D[Resolves to os.system]
    D --> E[REDUCE: code execution]

Preconditions / Affected

• Exposure: the paper surveyed 10,023 popular Hugging Face repos created 2025-07-01 to 2026-06-30 with over 1,000 monthly downloads. It found 9,544 Pickle-based PTMs across 929 repos (9.3%), including repos from NVIDIA, IBM and OpenAI.
• Hugging Face's own PickleScan labels on 8,859 suspected-Pickle files: 5,840 safe, 2,998 caution, 21 unsafe.
• Existing scanners: PickleScan flagged 177 of 706 benign models (25.1%) as false positives. The best existing scanner reached F1 0.776 on PickleBench.

What to do

• If you gate model ingestion on a Pickle scanner: check whether it tracks copyreg registry mutation before EXT* opcodes. The paper says existing scanners cannot faithfully track this.
• If you can, prefer Safetensors or GGUF. The paper treats them as data-based formats with stronger guarantees.
• To evaluate DITTO: it reports 100% scanning coverage, 0% false negatives, 0.7% false positives and F1 0.966 on PickleBench, with the zero-miss result holding across three runs. The fetched text names no release location, so check the paper itself for artifact availability.

Caveats

• Results come from the authors' own benchmark. The 92 malicious samples are "representative" attacks from prior work plus the authors' new extension registry attacks.
• The 0% false-negative rate is measured on that small set and is not evidence of general evasion resistance.
• The paper states the output is meant for human-in-the-loop auditing, not automatic allow/deny.
• The scanner's accuracy depends on the chosen semantic analyzer, and the fetched text does not say which one the default configuration uses.
• The fetched page cuts off partway through the motivation section. The evaluation details, ablation and the analyzer comparison were not available to me.