Revised instructions were associated with an increase in accuracy from 52% to 70% in a study of LLM-based legal information extraction submitted on November 2, 2026. But Tomáš Knap and colleagues also found that a model agreeing with itself was no guarantee that it had extracted the right answer.
The researchers present a pipeline for turning criminal verdicts into structured datasets. These decisions describe criminal behavior that can support quantitative legal research, but the attributes and extraction rules needed to organize that information cannot always be settled in advance.
The approach combines attribute exploration, text span extraction and value normalization, with legal experts helping refine the annotation instructions that guide the process.
Testing LLM-based legal information extraction on traffic offenses
Two specific offenses were examined in the study: driving under the influence (DUI) and driving despite a ban (DDB). Its findings connect instruction clarity with extraction accuracy and agreement.
In DUI cases, the share of correctly extracted values rose from 52% to 70% after instruction revisions. Human annotators also agreed more often after that refinement.
The repeated-run results showed a different distinction. Outputs from the same model matched one another more closely than they matched the gold annotations used for comparison. The researchers’ finding separates consistency from correctness: producing the same result repeatedly did not establish that the result was right.
Disagreements among annotators and among models helped expose unclear instructions. In this legal information extraction pipeline, those differences provided evidence of where the guidance needed clarification.
For DDB, instructions specified fixed categories and output formats. Agreement was high across most attributes in that evaluation.
Article produced with the assistance of artificial intelligence and reviewed by the editorial team.
