BLiMPPI: Benchmark of Linguistic Minimal Pairs for Polarity Items

What it claims

BLiMPPI is a multilingual forced-choice benchmark of 12,866 negative polarity item (NPI) minimal pairs in 33 languages, built mostly from UD v2.18 treebanks. Each pair perturbs a naturally occurring sentence, so the data has more syntactic and lexical variety than templated suites. For comparison, BLiMP's 7,000 NPI pairs all test English ever. The set combines classic NPIs with Negative Concord Items such as Polish nikt and Turkish hiç kimse, and it is heavily skewed by language:

• Russian has 4,199 pairs, Dutch 1,771, Czech 1,213 and Turkish 813.
• Indonesian has 6, Hungarian 13 and Serbian 24.

By typology, strict negative concord languages contribute 8,171 pairs, other NC languages 1,351 (plus Catalan 220 and Spanish 272 as non-strict NC), and NPI languages 2,852.

How it works

flowchart TD
  A[UD v2.18 treebank] --> B{Linguistic NPI list?}
  B -- yes --> C[Use list, incl. multi-word items]
  B -- no --> D[Discover: DET/PRON/ADV lemmas, freq ≥5, licensing-ratio cutoff]
  C --> E[Detect licenser, climb into matrix clause if needed]
  D --> E
  E --> F{Overlapping licensers?}
  F -- yes --> G[Discard]
  F -- no --> H[Perturb: reinflect / delete / swap licenser / undo inversion]

• Treebank selection. A treebank must have at least 100k words (Hungarian is an exception). The language must have stable, symmetrical negation that is recoverable from UD, and it must be a living language. French is excluded because of its shifting ne…pas paradigm.
• NPI discovery is used only when no linguistic resource exists. The cutoff is set per language, and about 70% works for NC languages. The signal is weaker in NPI languages: only 56% of NPI occurrences fall in a detected licensing environment, against 78% for NCIs.
• Licenser detection. Licensers are found by UD relations and features (Table 2). For non-island complements such as ccomp and xcomp, detection walks up to the matrix clause, which is how "I don't think she has ever lied" is handled.
• Multiple NPIs. Sentences with more than one NPI (7.5% of pairs) are kept but flagged. They shift accuracy by about +0.9 points.
• External sources. English adds the NPI bank and Dutch comes from the Dutch NPI Lexicon. Both were already annotated.

Evidence

The evaluation covers 12 multilingual and 2 monolingual LMs from 8 families, plus n-gram baselines. It also tracks checkpoints of 7 monolingual models to see how NPI knowledge emerges during training. The fetched text gives no per-model accuracies or model names, only the direction of the result: multilingual models do worst on low-resource languages and on pairs that n-grams cannot solve.

The quality audit is the most useful number for anyone adopting the data:

• Scope: 638 judgements on 428 pairs across 17 languages.
• Flag rate overall: 9.6% of pairs did not carry the intended contrast.
• Flag rate by typology: 0–7% in NCI languages, but 10% for Danish, 20% for German and 30% for English. The authors blame polysemy, where the perturbed sentence ends up with a grammatical reading.
• Fixes: flagged items were corrected or dropped, and systematic problems were fixed by regenerating the data.
• Verification method varies: 7 languages were built with a native speaker or linguist. Several others (Belarusian, Bulgarian, Persian, Slovak, Slovenian, Catalan, Hebrew, Icelandic, Indonesian) were checked only by machine translation.

How to use it

The data is at https://github.com/arzuburcuguven/blimppi-data. Each language subset carries the licence of its source treebank, so check the repository's licence table before redistributing or using it commercially. Pairs are scored the standard way, by comparing LM probabilities of the two sentences.

Limits

• The Germanic subsets, including English, had the highest audit flag rates, and only a sample was audited. Treat absolute scores on German and English with caution.
• Several languages have too few pairs for per-language conclusions, notably Indonesian (6), Hungarian (13) and Serbian (24).
• Discovered NPI inventories are less reliable in NPI languages because polysemous items like any are both free-choice and NPI.
• Languages with asymmetric or unstable negation were excluded by design.
• The fetched text cuts off partway through the quality-control section, so the model results and analysis sections were not available here.