LexPilot: Benchmarking the Proactive Interaction Ability of Legal LLMs with Laypeople in Chinese Civil Case Consultation

What it claims

LexPilot is a Chinese legal benchmark from Harbin Institute of Technology and Tsinghua University (arXiv, October 2026). It measures whether an LLM asks a layperson for missing case facts before predicting a civil verdict, and how much that changes claim-level prediction accuracy. It holds 864 consultations: a complete and an underspecified multi-turn history for each of 432 first-instance judgments from China Judgments Online, with 12 judgments for each of 36 causes of action. Ground truth is the real court's verdict on each claim, labelled "Supported", "Partially Supported" or "Dismissed".

How it works

Evaluation splits proactive interaction into three sequential tasks:

flowchart LR
  H[Consultation history] --> LCA{LCA: facts sufficient?}
  LCA -- yes --> LJP[LJP: claim-level verdict]
  LCA -- no --> DCG[DCG: ask user for missing details]
  DCG --> LJP

Legal Completeness Assessment (LCA) decides whether the facts support a conclusion. Detail Completion Guidance (DCG) tests whether the model elicits the missing facts. Legal Judgment Prediction (LJP) scores the final verdict.

Construction has five steps. First, judgments are segmented into header, facts, reasoning and results. Second, focal points of dispute are taken from the court's reasoning, and the matching key facts are extracted. Third, plaintiff profiles are built. Fourth, paired dialogues are simulated LeCoDe-style from expert-written seeds, one seed per cause of action. Fifth, the dialogues are anonymized. Paired histories have equal turn counts and differ only in whether dispute-critical details are disclosed. Two legal experts with law master's degrees review all dialogues. They add facts back into complete histories and strip residual critical details from underspecified ones.

The baseline method is LexAwN, a legal adaptation of the AwN proactive-interaction method (Wang et al., 2025). It is run across general and legal LLMs.

Evidence

Two pilot studies motivate the benchmark. In the first, 16 undergraduates wrote consultations from memory after viewing a case for two minutes: 8 were law majors and 8 had no legal background. Lay participants covered 68.1% of expert-annotated key details, against 86.6% for law majors, at similar lengths (157.6 vs 174.2 tokens). In the second, three advanced LLMs were given underspecified consultations. They flagged the missing information only 4.2% to 7.3% of the time. They predicted a verdict directly 34.4% to 41.3% of the time and gave generic advice 19.7% to 32.0% of the time.

On the benchmark, proactive interaction generally improves prediction on underspecified consultations. It still trails the fully specified setting by 23.84% relative on average. The authors report that performance varies with case complexity and with the type of omitted detail.

How to use it

Code and data are on Hugging Face at Medivvv/LexPilot (https://huggingface.co/datasets/Medivvv/LexPilot). The page states no licence.

Limits

The fetched text ends at the statistics table. Per-model scores, the LCA and DCG metrics, the model list and the bottleneck analysis are not visible here. All consultations are LLM-simulated from seeds and then expert-edited, so none are real user dialogues. Coverage is Chinese civil first-instance cases only. The lay-omission pilot used undergraduates recalling cases from memory, not people describing their own disputes.