Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System

What it claims

Closed multiple-choice accuracy on Colombian law is a good ranking signal but a poor absolute one. Across 15 proprietary and open-weight models, closed-question accuracy and free-text factual correctness are strongly rank-correlated (Spearman ρ = 0.94). A cheap multiple-choice screen therefore orders models correctly while overstating how reliable they are. The free-text ranking holds under two independent checks: a rubric-based LLM judge and blind human expert scoring both reproduce it (ρ ≥ 0.88). Reliability also varies systematically by legal area and follows an inverted-U across question complexity. The fetched text does not say which end is worse or give per-area numbers.

How it works

The benchmark has 1,042 items: 305 closed, 682 semi-open and 55 open-ended.

• Closed: four options, exactly one correct, each tied to a cited article or ruling.
• Semi-open: answers of up to 150 words, stratified by complexity:
• Low (258 items): normative existence, competent authority, legal hierarchy, basic definitions.
• Medium (236 items): sense of a ruling, temporal validity, precedent, requirements and exceptions.
• High (188 items): normative conflicts, balancing of principles, systematic interpretation.
• Open-ended: case problems with an expert reference answer written in IRAC form (Issue, Rule, Application, Conclusion). The model prompt does not mention IRAC, and scoring is anchored to the reference answer's content, not its structure.

Coverage by area runs from Constitutional (134 items) down to Markets (72), which covers competition, consumer, data and IP law. The open-ended set is thin per area: 2–9 items each.

Metrics:

• Closed items are scored by accuracy.
• Free-text items are scored with BERTScore, BLEU, ROUGE, and RAG-style answer relevancy and correctness.
• Spanish readability indices are also computed, but they describe style, not correctness.

Different metrics cover different subsets of the benchmark.

Construction: Items were collected on a purpose-built platform, Horizontes Conectados, with roles for student authors, a legal assistant, expert evaluators and a technical team.

• 1,084 questions were received: 415 from five Universidad de los Andes law-student workshops (Nov 2025–Mar 2026) and 669 from the legal team and students working individually.
• Every item went through the same review sequence: primary review, expert evaluation, then a committee decision.
• 42 items were archived during review, leaving the final 1,042.

Evidence

• Closed accuracy: 0.577 to 0.905 (best: Gemini 3.1 Pro).
• Free-text factual correctness: at most 0.45 for any model.
• Relevancy vs. correctness: ρ = −0.46.
• Ranking agreement: ρ = 0.94 between closed and free-text rankings; ρ ≥ 0.88 between the LLM judge and humans.

Prior baselines cited:

• Manrique et al. tested 458 bar-exam multiple-choice items from Bolivia, Colombia and Puerto Rico; the best model reached 79.7%, with criminal and administrative law the weakest areas.
• A 2024 Colombian study of ChatGPT-3.5, ChatGPT-4 and Gemini Pro found no model acceptable for practical use.
• U.S. RAG legal tools (Lexis+ AI, Westlaw) still erred 17–33% of the time.

How to use it

The authors state they release the benchmark construction pipeline for reproducible evaluation. The fetched text gives no repository link or licence, and does not explicitly say the 1,042 items themselves are released. Check both before planning to run it.

If you only need to rank candidate models for Spanish/Colombian legal work, the closed subset is a defensible cheap proxy. Do not read its absolute scores as deployment reliability.

Limits

• The fetched page cuts off in Section 4.3. The results sections, the per-model table, the per-area and complexity breakdowns, the list of 15 models, and the authors' stated limitations were not visible. Numbers above come from the abstract and the method sections.
• Only 55 open-ended items exist, so per-area open-ended conclusions rest on very few questions.
• Citation correctness ("about half" of cited norms are right) is measured by the LLM judge, which was validated against human experts. It is not fully human-scored.
• The benchmark covers Colombian civil law only, and the authors argue findings do not transfer across jurisdictions or languages.
• Grounding answers in authoritative sources is proposed as a "promising path"; the visible text does not show it was tested.