Accuracy in LLM legal information extraction rose from 52% to 70%

What it claims

Tomáš Knap and colleagues describe a pipeline that turns criminal verdicts into structured datasets for quantitative legal research. The study was submitted on 2 November 2026. It covers two traffic offenses: driving under the influence (DUI) and driving despite a ban (DDB). The pipeline is aimed at the case where the attributes to extract, and the rules for extracting them, cannot be fixed in advance. Its central finding is that clearer annotation instructions went with higher extraction accuracy and higher agreement. The page says the revisions were "associated with" the accuracy gain, not that they caused it.

How it works

The pipeline has three stages: attribute exploration, text span extraction and value normalization. Legal experts revise the annotation instructions that guide these stages. Disagreement is used as a diagnostic. Where human annotators disagree with each other, or models disagree with each other, the page treats that as evidence that the instruction for that attribute is unclear and needs revision.

flowchart LR
  I[Annotation instructions] --> P[Explore attributes, extract spans, normalize values]
  P --> M[Model outputs]
  H[Human annotations] --> D{Disagreement?}
  M --> D
  D -- yes --> E[Legal experts revise instructions]
  E --> I
  D -- no --> S[Structured dataset]

Evidence

For DUI, the share of correctly extracted values rose from 52% to 70% after the instructions were revised. Agreement between human annotators also rose after the revision, but the page gives no figure for it. For DDB, the instructions specified fixed categories and output formats from the start, and agreement was high across most attributes. The page gives no number for this and does not say which attributes fell short. In repeated runs, outputs from the same model matched each other more closely than they matched the gold annotations.

Limits

The page is a short news article, produced with AI assistance and reviewed by an editor. It does not link the paper. It does not name the models, the number of verdicts, the attributes extracted, the jurisdiction or any per-attribute results. It also does not say whether code or data are released. The DUI accuracy figures come from one offense type, and the DDB result has no figure attached. Because self-consistency did not track correctness, agreement across repeated runs of one model cannot stand in for a check against gold annotations.