Small Edits, Shifting Decisions: Benchmarking LLM Accuracy and Reliability on French Input Variants

What it claims

The benchmark tests French LLM inputs under three surface variations: apostrophe omission ("jai" for "j'ai"), diacritic removal ("ecole" for "école") and AZERTY keyboard typos. It has 4,200 original/low/high triplets drawn from seven French task datasets. Across six Qwen3 and Gemma-3 models (4B to 32B), accuracy falls and Brier score and NLL rise with perturbation intensity. The summary does not say how scale interacts with this. Larger models lose less accuracy at high intensity, but their Brier score and NLL increases do not consistently shrink. A bigger model therefore holds its answers better without its probabilities degrading any less.

How it works

Seven sources supply the items, each with 200 base items per perturbation type and no overlap of base items between types:

• MARC review rating (5 choices)
• FrenchNLI (3)
• Belebele reading comprehension (4)
• French HellaSwag (4)
• the mathematics subset of Global-MMLU (4)
• HistoiresMorales moral judgment (2)
• MCL-WiC word-in-context (2)

Target edit rates are 2.5% and 5% of words for apostrophe omission, and 5% and 10% for diacritics and typos. The realized means are 2.50/5.10, 4.99/9.96 and 5.00/10.14. Variants are strictly nested: the high variant keeps every low edit and adds at least one more. Edit counts use random rounding with fixed seeds.

Some parts of each item are never edited. These are reference answers, option order, numbers, structural task markers, and the target words in word-sense tasks. Predefined lexical collisions are excluded.

Scoring is zero-shot. Each answer candidate gets its average token log-probability, and a softmax across the candidates turns these into probabilities. The metrics are accuracy, Brier score, NLL and ECE (10 equal-width bins). For temperature scaling, one temperature per model is fitted only on the original inputs of the held-out split.

Evidence

The data was validated in two stages:

• Model screening. DeepSeek-V4-Pro and Gemini-2.5-Pro each checked every triplet for answer validity.
• Human review. Three CEFR B2+ annotators from the language service provider Lanbridge reviewed 26 flagged triplets plus a 210-triplet stratified audit. Krippendorff's α was 0.969 initially and 1.000 after the guidelines were clarified. Four triplets were replaced.

The splits are 840 held-out triplets (40 per dataset and perturbation type) and 3,360 test triplets. Splits are made at the source-group level so that shared passages do not leak between them.

Figure 1 shows the failure pattern. Qwen3-14B answers C (correct) at 59.9% on the original input, still answers C at 47.2% at low intensity, and flips to D at 36.7% at high intensity. Confidence in the correct answer drops before the prediction flips.

The fetched text ends partway through the experimental setup. Per-model accuracy, Brier and NLL tables, and the size of the calibration-plus-restoration gains, are not available here.

How to use it

The protocol only works for tasks with fixed candidate answers, because it needs per-candidate log-probabilities. Open-ended generation is out of scope.

The held-out/test split is built for fitting a post-hoc temperature on clean data and checking it on perturbed data. Teams with a deployed French classifier or multiple-choice pipeline can reuse it that way.

The fetched text gives no release location, licence or code for the benchmark.

Limits

• Only two model families are tested, Qwen3 and Gemma-3, at 32B or below.
• Each base item gets one perturbation type at a time. Combined noise is not tested.
• Typos are synthetic AZERTY-neighbour substitutions, not real user errors.
• The finding that keyboard typos hurt most is stated only for the predefined high-intensity settings. Typos are run at 10% edit rate while apostrophes are run at 5%, so the types are not compared at equal intensity.
• Annotators were B2+ rather than native speakers.