We’ve been analyzing how people are using LLMs for legal and compliance tasks (GDPR, AI Act, etc.).
Summary
Analysis of LLM usage in legal and compliance tasks reveals that models often produce confident but unverifiable citations, raising questions about reliable legal grounding for AI outputs.
Similar Articles
Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases
This small-scale study evaluates LLM reasoning in legal case forecasting using European Court of Human Rights cases, finding that models produce structurally complete but substantively shallow analyses and that LLM-based evaluators align weakly with human annotators.
Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems
This paper proposes techniques that combine formal methods (Linear Temporal Logic) with LLMs for auditing, monitoring, and intervening in AI systems to ensure compliance with behavioral constraints, showing that even small-model labelers can match frontier LLM judges in detecting violations.
What happened to the issue of companies running out of training data for LLMs?
The article revisits the earlier concern that human-generated training data for LLMs would run out, questioning whether the issue has been resolved or remains a problem given the continued improvement of AI models.
Using LLMs
The article reflects on the limited use and understanding of LLMs such as ChatGPT among personal acquaintances, raising questions about widespread AI tool adoption.
The problem with LLMs as judges
The article highlights the inefficiency and rationalization issues with using LLMs as judges, introduces Jev as a tool that leverages structured data for more reliable results, and questions the potential for open-source alternatives.