Is this Citation on Point?
Summary
This paper evaluates large language models' ability to verify citation support in legal documents, finding that while models detect wrong-case citations effectively, they struggle with pinpoint page references, often confusing topical relevance with precise support.
View Cached Full Text
Cached at: 08/17/26, 11:48 PM
Paper page - Is this Citation on Point?
Source: https://huggingface.co/papers/2608.12571 Published on Aug 12
·
Submitted byhttps://huggingface.co/0xe69756
Apurvon Aug 17
Abstract
Large language models reliably detect incorrect case citations but frequently miss wrong pinpoint page references, conflating topical relevance with precise legal support.
In 2023, a New York judge sanctioned two attorneys in Mata v. Avianca for filing a brief with hallucinated citations generated by ChatGPT. Such failures are largely caught by database lookups; the harder problem is detecting citations that point to real cases but do not support the propositions for which they are offered -- a failure mode that existing evaluations of LLMs for legal use cases largely overlook. In this paper, we studyproposition-level citation support verificationthroughcontrolled perturbationsof real legal citations obtained from two legal corpora, either replacing the cited case or changing only thepinpoint pagewithin the same case. We evaluate fourteen model configurations on the resulting examples. Models catch 93-100% ofwrong-case corruptions. They catch only 37-61% ofwrong-pinpoint corruptionson court opinions and 52-83% on legal briefs. When models fail to catchwrong-pinpoint corruptions, they accept the citation based ontopical overlaprather than page-level support. Scale and extended reasoning narrow the gap but do not close it: GPT-5.4 with highreasoning effortstill misses 40% of pinpoint mismatches on court opinions and 18% on briefs. Prompting the model to verify support at the cited page improves recall, but it also raises thefalse positive rate. Recognizing the right legal topic and verifying support for the cited proposition are distinct capabilities, and current models conflate them.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2608\.12571
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.12571 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.12571 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.12571 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Large Language Models for Citation Function Classification
This paper presents a comprehensive evaluation of five large language models for citation function classification, achieving new state-of-the-art results on the ACL-ARC dataset with a fine-tuned Falcon 7B model. It also introduces the AC3 dataset, which includes a seven-category annotation scheme distinguishing neutral acknowledgments from evaluative stances.
Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution
This paper benchmarks 8 LLM judges for citation quality in deep-research systems, finding that cheaper models remain competitive with frontier models on source relevance and factual support, but differ in directional bias which matters for RL training.
CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence
CiteVQA is a benchmark for document vision-language models that evaluates both answer correctness and citation of supporting evidence, revealing widespread attribution hallucinations where models provide correct answers but cite wrong regions.
On the Role of Citations in Preference Data
This paper investigates the role of citations in human and LLM preferences for scientific question answering, finding that humans prefer diverse citations but fewer overall, while LLMs exhibit stronger citation-related preferences despite lacking source access.
Local LLM for legal-document adaptation keeps hallucinating citations with total confidence — grounding/model/pipeline ideas?
A user reports that a local LLM hallucinates citations with high confidence when adapted for legal documents, and seeks advice on grounding, model, or pipeline ideas to mitigate this issue.