Legal Guidance in the Wild: How Users Seek Legal Help in Real-World LLM Conversations

Hugging Face Daily Papers Papers

Summary

The paper studies how users seek legal guidance in real-world LLM conversations, introducing a taxonomy, annotating 2,122 WildChat conversations, and releasing a 200-conversation evaluation set along with trained classifiers and public resources.

People ask conversational AI models about their legal rights, obligations, procedures, and personal situations that may have legal consequences. To study this behavior, we first need to distinguish requests for legal guidance from other legally themed discussions and unrelated conversations. We contribute a multi-axis taxonomy of legal queries capturing whether legal guidance is sought, how a request is formulated, and which legal domains it concerns; apply it to 2,122 naturally occurring WildChat-1M conversations; and create a human-reviewed gold evaluation set of 200 conversations. All annotations, model checkpoints, and evaluation code are publicly available. In this enriched sample, 71.1% of conversations containing legal content seek personalized or practical guidance. Rights-and-options advice is the most common primary request type among legal-content conversations, and 25.3% of guidance-positive conversations span multiple legal topics. The gold evaluation set supports a two-stage comparison of ModernBERT-base and ModernBERT-large against a TF-IDF cascade. Our framework supports large-scale studies of legal guidance in conversational AI, and the public release of the annotated data facilitates reproducible research and future model development.
Original Article
View Cached Full Text

Cached at: 10/21/26, 07:05 AM

Paper page - Legal Guidance in the Wild: How Users Seek Legal Help in Real-World LLM Conversations

Source: https://huggingface.co/papers/2610.25159 We study how people seek legal guidance in real-world LLM conversations. We introduce a taxonomy, annotate 2,122 WildChat conversations, and release a human-reviewed evaluation set of 200 conversations. We also train and evaluate smaller classifiers to detect guidance-seeking requests and classify their legal topics.

The datasets, model checkpoints, and evaluation code are publicly available. We hope these resources support further research into legal guidance seeking in conversational AI.

Happy to discuss the work and hear your thoughts!

Similar Articles

DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation

arXiv cs.CL

DLawBench is a new benchmark for evaluating large language models in multi-turn legal consultation, covering Chinese and US law with four client types. Experiments show significant room for improvement, with the best model achieving only 0.562 on legal reasoning.