Reversible Pseudonymisation of Documents Sent to External LLMs: an Open Bilingual Benchmark and an Engine

What it is

A benchmark and engine paper (arXiv 2610.34012v1, October 2026) from Vladimir Beskorovainyi of Besk Tech. It tests reversible pseudonymisation of documents sent to external LLMs in Russian and UK English. The engine, pii-roundtrip (release 0.5), combines two of the tested tools. The page text names pii-guard as one of them and GLiNER Guard as the other. Code, data, predictions and raw LLM answers are released.

How it works

The benchmark covers five open tools and a checksum-validated regex baseline:

• pii-guard (rules plus a ruBERT classifier, Russian only)
• GLiNER Guard Omni
• OpenAI privacy-filter
• Presidio (with Russian identifier recognisers added by the author)
• Prompt Anonymizer (English only)

Four thresholds were set before any run:

• Recall of at least 99% on structured identifiers.
• Recall of at least 95% on names, addresses and organisations.
• Leaks in at most 2% of documents.
• At least 99% of placeholders restored after an LLM answer.

Two more were added later: a strict count of partly masked values (at most 2%), and 99% restoration of name placeholders. A threshold counts as met only if the whole 95% interval lies on the right side.

Existing tools are judged on external datasets, never on their own authors' data:

• Russian: hivetrace/pii-bench and the redmadrobot pii_benchmark.
• UK: Nemotron-PII, including a subset where the author injected valid NHS numbers, NI numbers and IBANs.

The paper shows why this rule matters. GLiNER Guard finds 87.1% of structured identifiers on its home dataset and 70.8% on the other Russian one.

Leaks are matched across number separators and across all Russian case forms of surnames. Tools lacking a type, such as organisations for pii-guard and OpenAI privacy-filter, are charged with those leaks. A leak rate also rises with document length: held-out documents carry about 30 (Russian) and 26 (UK) countable values.

Results

• On external datasets no existing tool meets every threshold. In UK English the identifier and leak thresholds fail. In Russian the identifier threshold is borderline.
• On held-out set 7 (1,000 documents per language), the best existing tools leave a whole value visible in 61% (Russian) and 58% (UK) of documents.
• The engine does so in 0.3% [0.1; 0.9] and 0.2% [0.1; 0.7]. Any part of a value stays visible in 0.8% and 0.2%.
• The engine meets all detection thresholds in both languages.
• UK restoration reaches at least 99% after 760 answers from two local LLMs.
• Russian name restoration is 94.6% [93.6; 95.6], below 99%. The cause is pii-guard's inflection, which produces wrong surname forms.

Caveats

• The engine was developed over six earlier held-out sets from the same generator, and set 7 uses that generator too. The hard formats were chosen by the author and the engine was built for them.
• The author built the engine and ran the benchmark. Pre-registration of sets 5 to 7 is public, but the initial thresholds rest on a private development history.
• All data are synthetic and real documents were not tested.
• The UK verdict for GLiNER Guard, and for the engine built on it, is in-distribution, because GLiNER Guard trained partly on Nemotron-PII. Of the UK identifiers, 83% were injected by the author's generator.
• Evaluation rules changed several times after data had been seen. Set 6 initially overstated Russian restoration.
• The metric counts literal values only, not quasi-identifiers or inference. A partly masked address does not count as a leak under the main metric.
• The fetched text ends partway through the metrics section. Round-trip tables for other tools and the appendices were not available.