Reversible Pseudonymisation of Documents Sent to External LLMs: an Open Bilingual Benchmark and an Engine

What it claims

The engine is pii-roundtrip (version 0.5). It combines the two strongest tools in the benchmark: GLiNER Guard Omni and pii-guard. It replaces personal data with placeholders before an external LLM call and restores them locally in the answer. On external datasets, none of the five existing tools meets all of the author's predefined thresholds. In UK English, the structured-identifier recall and leak thresholds fail. In Russian, identifier recall is borderline. The engine meets all detection thresholds on the final held-out set 7 in both languages. It fails Russian name restoration at 94.6% [93.6; 95.6] against a 99% threshold, mainly because the surname inflection it takes from pii-guard produces wrong forms.

How it works

The benchmark covers pii-guard (rules plus a ruBERT classifier, Russian only), GLiNER Guard Omni, OpenAI privacy-filter, Presidio 2.2.364 and Prompt Anonymizer 0.3.3, plus a checksum-validated regex baseline. Each tool runs per its own documentation, with pinned commits. Presidio received added Russian recognisers for passport, SNILS, INN, bank account and Mir cards. pii-guard and privacy-filter have no organisation type. Prompt Anonymizer also lacks NHS and NI numbers. Leaks of unsupported types still count against each tool.

Four rules shape the comparison:

• No home dataset. No tool gets a verdict on its own authors' dataset. GLiNER Guard finds 87.1% of structured identifiers on its home dataset and 70.8% on the other.
• Thresholds. Recall of at least 99% on structured identifiers and at least 95% on names, addresses and organisations. Leaks in at most 2% of documents. At least 99% of placeholders restored. A threshold counts as "met" only when the whole 95% interval clears it.
• Leak matching. Numbers are matched across spaces, hyphens, dots and slashes. Russian surnames are matched in all grammatical cases.
• Round trip. Two local LLMs answer prompts, and model-invented placeholders are counted separately from tool-issued ones.

Evidence

• Russian external data: hivetrace/pii-bench (1,810 short texts, Apache 2.0) and redmadrobot pii_benchmark (2,841 sentences, MIT).
• UK external data: the Nemotron-PII test split (CC BY 4.0). It has 801 texts as they are, plus 2,333 texts with injected synthetic NI numbers, NHS numbers and IBANs. 83% of the 3,249 UK identifiers were injected by the author's generator.
• Held-out set 7: 1,000 generated contracts, letters, HR documents and discharge summaries per language, with about 30 (Russian) and 26 (UK) threshold values per document. About 40% of documents are made noisy. Within noisy documents, identifiers appear in hard formats (in words, dotted, or with a typo that breaks the checksum).
• Leak results on set 7: the best existing tools leak a whole value in 61% (Russian) and 58% (UK) of documents. The engine leaks a whole value in 0.3% and 0.2%, and any part of a value in 0.8% and 0.2%.
• Engine round trip: UK restoration reaches at least 99% after 760 answers.
• Stability: existing tools' leak rates agree within 3.7 points across held-out sets 4 to 7.

How to use it

The code, generator, held-out sets, per-document predictions and raw LLM answers are released. The generator reproduces every set byte for byte. Set 7's files, hashes, decision rules and stopping rule were committed publicly (commit 13c42b8, 23 October 2026) before any run. The tool configurations above (GLiNER threshold 0.4 with 1,200-character windows; Presidio score threshold 0.3) are a usable starting point. The page gives no install command or engine licence.

Limits

• All data are synthetic, and real documents were not tested.
• The engine was developed on the same generator across held-out sets 1 to 6, and its hard formats were chosen by the author.
• Both Russian external datasets are home datasets of the engine's components, so the engine's verdicts rest only on generated documents.
• GLiNER Guard was trained partly on Nemotron-PII, so the UK result for GLiNER Guard and the engine is in-distribution.
• Evaluation rules were changed several times after data had been seen.
• The author built the engine and ran the benchmark.
• The leak metric covers literal values only, not quasi-identifiers or attribute inference.
• A partly masked address does not count as a leak under the main metric.