We’ve been analyzing how people are using LLMs for legal and compliance tasks (GDPR, AI Act, etc.).
Summary
Analysis of LLM usage in legal and compliance tasks reveals that models often produce confident but unverifiable citations, raising questions about reliable legal grounding for AI outputs.
Similar Articles
Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems
This paper proposes techniques that combine formal methods (Linear Temporal Logic) with LLMs for auditing, monitoring, and intervening in AI systems to ensure compliance with behavioral constraints, showing that even small-model labelers can match frontier LLM judges in detecting violations.
What happened to the issue of companies running out of training data for LLMs?
The article revisits the earlier concern that human-generated training data for LLMs would run out, questioning whether the issue has been resolved or remains a problem given the continued improvement of AI models.
DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation
DLawBench is a new benchmark for evaluating large language models in multi-turn legal consultation, covering Chinese and US law with four client types. Experiments show significant room for improvement, with the best model achieving only 0.562 on legal reasoning.
Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning
This survey examines recent progress in medical LLMs, presenting a dual-view approach that connects clinical practice with computational methods, and introduces a benchmark dataset for evaluating medical reasoning capabilities across 18 state-of-the-art models.
If you use LLMs for work that matters, how do you decide when to trust the output?
A conceptual guide on deciding when to trust LLM outputs in high-stakes professional contexts like legal, clinical, and financial work, emphasizing the need for critical evaluation skills.