Tag
This paper proposes automatically auditing answer correctness and legal authority grounding jointly in legal benchmarks, showing that LLMs often produce correct answers while citing wrong governing provisions under ordinary prompts.
The article discusses a Hugging Face experiment where an automated loop rewrites only the code (harness) around a frozen model, raising its benchmark score from 0% to near Sonnet 4.6 at lower cost, demonstrating that many benchmark failures stem from the harness, not the model itself.
Harvey partnered with Applied Compute to train a legal agent, optimizing the agent stack and post-training the GLM-5.1 model using reward signals from their Legal Agent Benchmark.
TW-LegalBench is a benchmark for evaluating large language models on Taiwanese legal understanding, including over 16,000 multiple-choice questions, 117 essay questions, and 14,000 legal judgment prediction instances. Results show top models exceed the passing threshold for lawyers but fall short for judges, highlighting challenges in reliable legal text generation.
Researchers release LegalBench-BR, the first public benchmark for evaluating LLMs on Brazilian legal text classification, showing LoRA-fine-tuned BERTimbau dramatically outperforms GPT-4o mini and Claude 3.5 Haiku.
Harvey AI released Harvey LAB, an open-source benchmark for evaluating LLM agents on realistic legal work, featuring a dataset of 1,671 tasks across 24+ practice areas and an execution harness.