When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning

# When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning
Source: [https://arxiv.org/html/2608.14610](https://arxiv.org/html/2608.14610)
Yiqian Huang1,Shuyuan Zheng2,Qianying Liu3,Shaowen Peng4, Yuntao Kong5,Kotaro Funakoshi1,Chuan Xiao2,Manabu Okumura1,Yang Cao1 1Institute of Science Tokyo2Osaka University3NII LLMC 4Nara Institute of Science and Technology5Center of Juris\-Informatics, ROIS\-DS h1k@lr\.first\.iir\.isct\.ac\.jp, zheng@ist\.osaka\-u\.ac\.jp

###### Abstract

Legal reasoning tasks such as legal judgment prediction \(LJP\) require identifying the temporally correct version of the law governing a case—a capability we term*temporal applicable\-law determination*\. However, whether large language models \(LLMs\) can reliably perform this task remains unexplored\. In this paper, we construct a benchmark to evaluate LLMs on temporal applicable\-law determination, and systematically investigate why they fail at temporal legal reasoning\. Our experiments reveal four key findings\. First, LLMs exhibit a strong bias toward applying the most recently enacted law, regardless of when the legally relevant facts occurred\. Second, this bias does not stem from an inability to understand that laws have temporal scope, nor from a lack of knowledge about historical statutes\. Third, we provide behavioral evidence that reinforcement\-learning\-shaped explicit reasoning may be a key mechanism: while improving general reasoning ability, it reduces the diversity of reasoning paths, causing models to converge on applying the current law\. Fourth, this produces a counterintuitive inverse relationship: models with stronger general reasoning ability tend to perform*worse*on temporal legal reasoning\. Our findings offer concrete guidance for future work on improving LLM performance in temporally grounded legal reasoning111Our code and data will be made available upon acceptance\.\.

When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning

Yiqian Huang1, Shuyuan Zheng2††thanks:Corresponding author\., Qianying Liu3, Shaowen Peng4,Yuntao Kong5,Kotaro Funakoshi1,Chuan Xiao2,Manabu Okumura1,Yang Cao11Institute of Science Tokyo2Osaka University3NII LLMC4Nara Institute of Science and Technology5Center of Juris\-Informatics, ROIS\-DSh1k@lr\.first\.iir\.isct\.ac\.jp, zheng@ist\.osaka\-u\.ac\.jp

## 1Introduction

The rapid advancement of large language models \(LLMs\) has spurred growing interest in automating legal reasoning, with significant implications for judicial efficiency and legal accessibilityCuiet al\.\([2023](https://arxiv.org/html/2608.14610#bib.bib16)\); Huanget al\.\([2023](https://arxiv.org/html/2608.14610#bib.bib17)\)\. Researchers have explored LLM\-based approaches across a wide spectrum of legal tasks, including legal judgment prediction \(LJP\)Luoet al\.\([2017](https://arxiv.org/html/2608.14610#bib.bib18)\), legal fact predictionLiuet al\.\([2025](https://arxiv.org/html/2608.14610#bib.bib21)\), court opinion generationLi and Zhang \([2021](https://arxiv.org/html/2608.14610#bib.bib31)\), and contract analysisHendryckset al\.\([2021](https://arxiv.org/html/2608.14610#bib.bib30)\)\. These efforts have demonstrated that LLMs can serve as powerful tools in the legal domain, motivating broader deployment in legal practice\.

A critical yet underexplored dimension of legal reasoning is its inherently*temporal*nature\. The governing principle of*non\-retroactivity*dictates that legal events should generally be adjudicated under the law in force at the time the relevant conduct occurredBowen \([2005](https://arxiv.org/html/2608.14610#bib.bib19)\)\. Even when a statute has since been revised, its earlier version may remain operative for events predating the revision\. Misapplying the wrong temporal version is not a clerical error: it can fundamentally alter the legal characterization of conduct and ultimately the outcome of a case\. Accurate legal reasoning therefore requires models to identify which statutory version governs a given set of facts—a capability we term*temporal legal reasoning*\.

Despite its importance, only a handful of existing studies have considered the time dimension in LLM\-based legal analysis\(Baraleet al\.,[2025](https://arxiv.org/html/2608.14610#bib.bib22); Santosh and Vuong,[2025](https://arxiv.org/html/2608.14610#bib.bib28); Han and others,[2025](https://arxiv.org/html/2608.14610#bib.bib20)\)\. Among them, in LawShift,Han and others \([2025](https://arxiv.org/html/2608.14610#bib.bib20)\)found that even state\-of\-the\-art \(SOTA\) LLMs consistently fail to incorporate synthetic statutory amendments into their legal judgments, defaulting instead to predictions anchored in the current law\. However, they stop short of diagnosing*why*these failures occur, leaving the root cause unexplained\.

This paper investigates why LLMs fail to identify and apply the temporally correct version of the law\. Identifying the temporally applicable law is a necessary prerequisite for downstream legal reasoning tasks such as LFP and LJP: without first establishing which statutory version governs a case, any subsequent judgment prediction or legal analysis is built on an unreliable foundation\. To this end, we construct a benchmark for*temporal applicable\-law determination \(TALD\)*—requiring models to identify, given a case’s facts and the date of the relevant events, which statutory version governs the case\. Our empirical investigation yields four key findings\.First, LLMs exhibit a strong and systematic bias toward applying the most recently enacted law, regardless of when the legally relevant facts occurred—directly explaining the failures observed in LawShift\.Second, this bias does not stem from ignorance of prior statutory versions, nor from a failure to understand that laws have temporal scope\.Third, we identify reinforcement learning \(RL\) as a key mechanism: while it improves general reasoning ability, it simultaneously reduces the diversity of reasoning paths, causing models to converge almost exclusively on applying the current law\.Fourth, this produces a counterintuitive result: models with stronger general reasoning ability tend to perform*worse*on temporal legal reasoning\.

The contributions of this paper are as follows\.\(1\)To the best of our knowledge, this is the first work to systematically diagnose the root causes of LLM failures in temporal legal reasoning\.\(2\)We introduce a benchmark for temporal applicable\-law determination, enabling controlled evaluation of whether LLMs correctly identify the temporally appropriate statutory version for a given case\.\(3\)Our findings suggest that reasoning\-oriented RL can reduce diversity in TALD\-relevant reasoning paths, causing models to converge toward the current\-law trajectory\.

## 2The Temporal Applicable\-Law Determination Task

### 2\.1Preliminary

Time is a fundamental dimension of legal reasoning\. Statutes and regulations are enacted, amended, and repealed at specific points in time, and their legal force is bounded by corresponding effective periods\. A foundational principle governing this temporal structure isnon\-retroactivity\(Kryvoi and Matos,[2021](https://arxiv.org/html/2608.14610#bib.bib29)\): newly enacted law generally applies only to facts and legal relationships arising after its entry into force, not to those already completed under prior law\. Consequently, determiningwhich versionof a law governs a given dispute is a prerequisite for any legally sound conclusion\.

Temporal applicability determination is a foundational prerequisite for virtually all legal reasoning\. Before a model can apply a statute to reach a legal conclusion, it must first identify the version of that statute that was in effect at the legally relevant time\. Applying an incorrect version—whether superseded or not yet in force—leads to legally erroneous results regardless of the quality of the substantive reasoning\. Temporal applicable\-law determination is therefore a necessary first step in the legal reasoning pipeline\.

### 2\.2Task Definition

Consider that we have a collection of statutesN=\{1,…,n\}N=\\\{1,\\dots,n\\\}, where each statutei∈Ni\\in Nmay have multiple versions due to temporal legal evolution\. Given a query\(F,Q\)\(F,Q\), whereFFdenotes the facts of a case andQQdenotes the legal question by the user, theTemporally Applicable\-Law Determination \(TALD\)task aims to identify a sequence of legal citations𝒚=\[y1,…,yn\]\\boldsymbol\{y\}=\[y\_\{1\},\\dots,y\_\{n\}\]whereyiy\_\{i\}denotes the correct version of statueiithat is legally applicable to answeringQQbased onFF\. Note that ifyi=−1y\_\{i\}=\-1, it means that statuteiiis not cited in the legal reasoning process\.

Let𝒚^=\{y^1,…,y^n\}\\hat\{\\boldsymbol\{y\}\}=\\\{\\hat\{y\}\_\{1\},\\dots,\\hat\{y\}\_\{n\}\\\}denote the sequence of statute versions predicted by the model in the TALD task\. The model performance on TALD can be measured by the TALD accuracy, defined by the matching accuracy between the ground truth𝒚\\boldsymbol\{y\}and the prediction𝒚^\\hat\{\\boldsymbol\{y\}\}:

A​C​C​\(𝒚^;𝒚\)=∑i∈N𝟏​\(yi=y^i≠−1\)\|\{i∣yi≠−1​or​y^i≠−1\}\|\.ACC\(\\hat\{\\boldsymbol\{y\}\};\\boldsymbol\{y\}\)=\\frac\{\\sum\_\{i\\in N\}\\boldsymbol\{1\}\(y\_\{i\}=\\hat\{y\}\_\{i\}\\neq\-1\)\}\{\|\\\{i\\mid y\_\{i\}\\neq\-1\\text\{ or \}\\hat\{y\}\_\{i\}\\neq\-1\\\}\|\}\.\(1\)The objective of TALD is therefore to maximize the TALD accuracy\.

## 3Analysis Setup

### 3\.1Research Questions

We investigate the capability of large language models \(LLMs\) on the Temporal Applicable\-Law Determination \(TALD\) task through three research questions \(RQs\)\.RQ1:Can LLMs correctly determine the temporally applicable version of law based on the temporal information present in case facts?This examines whether models can leverage temporal cues in case descriptions—such as the timing of legal acts or disputes—to select the correct statutory version\.RQ2:Do LLM failures on TALD stem from deficiencies in knowledge memorization?Specifically, we examine whether models lack the necessary knowledge of temporal applicability rules governing Chinese civil law\.RQ3:Do LLM failures on TALD stem from deficiencies in legal reasoning?Even when models possess relevant legal knowledge, they may fail to correctly apply it to determine the temporally applicable statute\.

### 3\.2Dataset Construction

To evaluate LLMs on the TALD task, we collect civil judgment documents from the China Judgment Online platform\.222[https://wenshu\.court\.gov\.cn](https://wenshu.court.gov.cn/)For each judgment, we require the model to identify the temporally correct version of applicable civil law given the case facts and the plaintiff’s claims\. Since Chinese civil judgments follow a standardized structure, we use regular expression matching to automatically extract case facts and plaintiff claims as TALD queries, and extract the cited civil law version as the ground\-truth label\. After automatic extraction, two legal experts conducted cross\-validation via random sampling to verify extraction accuracy\.

The resulting dataset contains 26,000 civil judgments, balanced between two splits: thepost\-Code split, comprising cases governed by theCivil Code of the PRCenacted in 2021, and thepre\-Code split, comprising cases governed by prior standalone civil statutes such as theProperty LawandContract Law\. Since the Civil Code essentially consolidates these prior statutes—its property and contract books correspond directly to the formerProperty LawandContract Law—we treat the Civil Code as a collection of multiple statutes, each aligned with a prior standalone statute, enabling more fine\-grained analysis of model performance on TALD\.

To address RQ2, two legal experts additionally construct 16 either\-or questions covering the legal provisions on temporal applicability of the Civil Code under Chinese law\. These questions assess whether LLMs possess the requisite knowledge of temporal applicability rules, providing a diagnostic lens on the source of model failures\.

### 3\.3Experimental Configurations

##### Models\.

We evaluate a broad range of state\-of\-the\-art reasoning models\. For proprietary models, we include GPT\-5\.4, Claude Opus 4\.6, Gemini\-3\.1\-Pro, DeepSeek\-V3\.2, and GLM\-4\.7\. For open\-source models, we evaluate Qwen3 \(235B, 80B, 30B\)\. We also include LegalOne\-8B, a domain\-specific legal reasoning model\. By default, all models are evaluated at their highest available reasoning effort setting\.

##### Test Data\.

To ensure robustness, we repeat each experiment four times and report average results\. In each run, we randomly sample 100 cases from the post\-Code split and 100 from the pre\-Code split, forming a balanced test set of 200 cases\.

## 4Analysis on RQ1

As the exploration of Research Question I, we conduct an initial test to examine whether the ability of most advanced reasoning Large Language Models to perform TALD\.

### 4\.1Results

#### 4\.1\.1Basic Results

Table 1:LLMs’ performance of TALD on both splits, evaluated with TALD accuracy metric \(Equation[1](https://arxiv.org/html/2608.14610#S2.E1)\)\.Table[1](https://arxiv.org/html/2608.14610#S4.T1)shows a sharp asymmetry between the two splits\. On the Old\-version split, every evaluated model scores below0\.250\.25, and seven out of eight score below0\.150\.15; Claude\-Opus\-4\.6 in particular drops to0\.0140\.014\. On the New\-version split the same models score between0\.700\.70and0\.840\.84\. The gap is not explained by the difficulty of the task in general: in both splits, recovering the applicable law requires the same kind of legal reasoning over the same case facts, and the gold answer in the New\-version split is simply the Civil Code book that subsumes the relevant area\. The asymmetry instead indicates a directional bias—LLMs preferentially apply the newest version regardless of when the legally relevant facts occurred\.

##### Finding 1\.

Advanced reasoning LLMs have directional failure in TALD\.They systematically perform poorly when an older version of the applicable law should govern the case, while performing competitively when the newer version applies\. This asymmetry is consistent with a strong default tendency toward citing the most recently enacted law\.

#### 4\.1\.2Fault\-Type Analysis

Table 2:Fault type statistics\. Old→\\rightarrowNew, New→\\rightarrowOld, Non\-ver respectively stand for the ratio of tested samples withOld\-to\-New,New\-to\-OldandNon\-Versionaltype of fault\.Results above show that models fail old\-version cases, but do not yet reveal whether the failure is version\-centric: do LLMs show basic awareness that the case concerns temporally versioned legal sources, and focus on determining them? We therefore analyze the wrong predictions on both splits\. For each missed gold target law in LLM’s test sample, we classify the fault into three categories:Old\-to\-New, where the gold version is a previous law but the model cites a new version;New\-to\-Old, where the reverse occurs; andNon\-Versional, where the model failed with an answer that contains only versionally unrelated laws\.

Table[2](https://arxiv.org/html/2608.14610#S4.T2)shows that only a small portion of failures are version\-irrelevant, whileOld\-to\-Newtype takes the majority\. Across models, this category accounts for the dominant share of fault mass, whileNew\-to\-Olderrors are rare\. This indicates that models often identify the relevant legal family or versional counterpart, but select the wrong temporal direction\. In other words, the models are not merely citing unrelated law\. They display basic awareness that the case concerns a temporally versioned legal source, yet their default selection is biased toward the newer version\.

##### Finding 2\.

Advanced reasoning LLMs’ failures in TALD are version\-centric\.They often locate the relevant statute family, but choose the newer version when the previous version should apply\.

## 5Experiments on RQ2

Table 3:Statutory knowledge memorization probe on failed previous\-version cases\. For each failed case, the model is asked to reproduce the content of both its predicted articles and the gold articles\. C\-F1, R\-L, and Edit denote character\-level F1, ROUGE\-L, and edit similarity with the official statutory text\.RQ1 shows that LLMs fail TALD in a directional way: they tend to apply newer law versions even when previous versions should govern the case\. We next examine what drives this failure\. We focus on two fundamental and mostly orthogonal capabilities of reasoning LLMs as remaining explanations: whether models lack the necessary legal knowledge, and whether their reasoning capability or reasoning policy is responsible for the failure\.

### 5\.1Legal and Temporal\-Effect Knowledge

##### Hypothesis 1\.

LLMs may fail TALD because they lack the relevant legal knowledge\.This may take two forms: they may not remember the content of older statutory provisions, or they may not know the temporal\-effect rules that determine whether old or new law should apply\.

##### Probe A: statutory\-text memorization\.

We first test whether models remember the relevant statutory texts\. On failed old\-version cases, we ask the same model to reproduce two sets of provisions: the articles from its own wrong prediction and the gold articles under the correct previous\-version law\. We then compare the generated content with the official statutory text using character\-level F1, ROUGE\-L, and edit similarity\.

##### Results\.

Table[3](https://arxiv.org/html/2608.14610#S5.T3)shows that most models can reproduce both predicted and gold provisions with high similarity to the official text\. Importantly, gold previous\-version articles are not systematically less memorized than the wrongly predicted articles\. For example, DeepSeek\-V3\.2 obtains 0\.963 character\-level F1 on gold articles and 0\.901 on predicted articles; Qwen3\-235B obtains 0\.902 on gold articles and 0\.776 on predicted articles\. This suggests that models often possess the old\-law textual knowledge needed for TALD, but still fail to select it in context\.

##### Probe B: temporal\-effect rule knowledge\.

Remembering statutory text is not sufficient for TALD; models must also know the legal principles governing temporal applicability\. We therefore construct a 16\-question multiple\-choice question \(MCQ\) probe based on expert\-designed questions about civil\-law temporal\-effect rules according to relevant judicial interpretation\. Each question asks the model to choose the legally correct option under a specified temporal\-effect condition\.

##### Results\.

Table[4](https://arxiv.org/html/2608.14610#S5.T4)shows that models generally perform well on this rule\-knowledge probe\. All models achieve accuracy above 0\.80, and several reach around 0\.90 or higher\. Even models that fail substantially on old\-version TALD cases can answer many abstract temporal\-effect questions correctly\. This indicates that TALD failure is not primarily caused by the absence of explicit knowledge about temporal\-effect doctrines\.

##### Finding 3\.

Knowledge deficiency is not the main factor behind TALD failure\.Models often possess both forms of knowledge required for TALD: the textual content of old/new statutes and abstract temporal\-effect rules\. Their failure, therefore, arises when applying this knowledge to concrete case facts\.

Table 4:MCQ accuracy performance on the Civil Code Temporal Effect Judicial Interpretation\.

### 5\.2Reasoning Capability

Having ruled out basic legal\-knowledge deficiency, we next examine whether TALD failure is caused by insufficient reasoning capability\. TALD requires models to connect legally relevant facts with the effective periods of candidate law versions\. A natural explanation is therefore that the task is simply too difficult for current models, and that stronger general reasoning should improve performance\.

#### 5\.2\.1Does stronger general reasoning help?

##### Hypothesis 2\.

A key factor behind TALD failure is insufficient general reasoning capability\.If this hypothesis holds, stronger reasoning configurations should perform better, especially on old\-version cases where temporal applicability must be inferred carefully\. Here, we use*general reasoning ability*to refer to the reasoning competence targeted by general\-purpose reasoning post\-training and broad reasoning benchmarks\. Then we derive the hypothesis below\.

##### Probe\.

We compare model configurations that differ in expected general reasoning strength, including instruct versus thinking modes, lower versus higher reasoning effort, and different model sizes within the same family\. All models are evaluated under the default prompt\.

##### Results\.

The "Default" colums of Table[5](https://arxiv.org/html/2608.14610#S5.T5)shows that stronger general reasoning does not consistently improve old\-version TALD\. In more than half of all LLMs, reasoning\-oriented configurations perform worse than less\-reasoning configurations\. For example, Qwen3\-80B drops from 0\.080 in instruct mode to 0\.028 in thinking mode\. Across model families the best performances are obtained by smaller or less\-reasoning configurations, e\.g\., Qwen3\-30B\-instruct \(0\.292\), GPT\-5\.4\-high \(0\.237\), Qwen3\-235B\-thinking \(0\.148\), rather than by the most reasoning\-heavy configurations of the largest models\.

This pattern contradicts the hypothesis that TALD failure is merely due to insufficient general reasoning ability\. If stronger reasoning naturally solved the task, reasoning modes and higher effort should consistently improve old\-version accuracy\. Instead, they can reinforce the same failure direction identified in RQ1: applying newer law versions when previous versions should govern the case\.

##### Finding 4\.

Stronger general reasoning does not solve TALD; engaging it more strongly often makes performance worse\.

#### 5\.2\.2Examine Task\-specific Reasoning Policy

The previous analysis shows that general reasoning ability do not result in better TALD performance\. A plausible explanation is that reasoning LLMs possess relevant reasoning capacity, but their default reasoning behavior is not aligned with the task\-specific version\-applicability policy required by TALD\.

##### Hypothesis 3\.

A key factor behind TALD failure is a misaligned task\-specific reasoning policy\.If this hypothesis holds, hints about temporal version applicability should improve TALD, especially for reasoning\-oriented configurations that have enough capacity to use the guidance\.

##### Probe\.

We evaluate models on the Old\-Version Split under three different system prompt settings, with othe settings same as Section[5\.2\.1](https://arxiv.org/html/2608.14610#S5.SS2.SSS1)\. The no\-hint setting uses a default prompt\. The weak hint and the strong hint respectively contains a brief reminder of legal principle for TALD and an detailed legal expertise instruction for TALD\. Full prompts are provided in Appendix[A](https://arxiv.org/html/2608.14610#A1)\.

##### Results\.

Table[5](https://arxiv.org/html/2608.14610#S5.T5)shows that temporal\-law hints substantially improve old\-version TALD for many reasoning\-oriented configurations\. DeepSeek\-V3\.2 reasoner improves from 0\.092 to 0\.436, GLM\-4\.7 thinking improves from 0\.080 to 0\.435, Qwen3\-80B thinking improves from 0\.028 to 0\.512, and Qwen3\-30B thinking improves from 0\.020 to 0\.395\. GPT\-5\.4 also improves from 0\.237 to 0\.457 under strong hint\. At the same time, the effect is not a uniform prompt bonus\. Some less\-reasoning configurations that already perform relatively well under no hint show limited or even negative changes, such as Qwen3\-30B instruct\. This suggests that hints help primarily when the model has reasoning capacity but needs guidance toward the correct temporal\-law policy\.

##### Finding 5\.

TALD failure is closely tied to task\-specific reasoning\-policy misalignment\.Domain\-expertise hints can redirect reasoning\-oriented models toward the correct temporal\-applicability policy, whereas stronger general reasoning alone does not guarantee improvement\.

ModelModeDefaultWeakStrongDeepSeek\-V3\.2chat0\.0130\.0280\.080reasoner0\.0920\.3940\.436Claude\-Opus\-4\.6low0\.0210\.1310\.185medium0\.0190\.1020\.143high0\.0260\.0990\.154max0\.0140\.1130\.156Gemini\-3\.1\-Prolow0\.1460\.5010\.439medium0\.1310\.5030\.489high0\.1350\.4740\.486GPT\-5\.4none0\.0210\.0870\.167low0\.0820\.4440\.456medium0\.1960\.4720\.486high0\.2370\.4160\.457
ModelModeDefaultWeakStrongGLM\-4\.7instruct0\.0130\.0320\.053thinking0\.0800\.3370\.435Qwen3\-235Binstruct0\.0350\.0350\.041thinking0\.1480\.3300\.323Qwen3\-80Binstruct0\.0800\.0780\.066thinking0\.0280\.3860\.512Qwen3\-30Binstruct0\.2920\.2640\.164thinking0\.0200\.3500\.395LegalOne\-8Breasoning0\.0930\.2800\.298

Table 5:Effect of temporal\-law hints on previous\-version cases\. Values are TALD accuracy\.

## 6Experiments on RQ3

RQ2 shows a counterintuitive phenomenon: stronger general reasoning ability does not naturally improve TALD, and can even drive LLMs towards more severe failure, with an unreasonably strong tendency to apply new versions of law\. However, this still leaves a mechanism question: why does the reinforcement of general reasoning result in a worse TALD task\-specific reasoning policy?

### 6\.1Hypothesis: Policy\-Entropy Collapse

We interpret this phenomenon through the lens of exploration and exploitation in reasoning\-oriented RL\. Entropy in reasoning policy conveys the diversity of LLM’s reasoning traces, serving as a useful signal of exploration in multi\-step reasoning\(Chenget al\.,[2025](https://arxiv.org/html/2608.14610#bib.bib3); Zhanget al\.,[2025](https://arxiv.org/html/2608.14610#bib.bib8)\)\. Prior work observes that RL training for reasoning LLM may suffer from policy\-entropy collapse, reducing exploratory behavior and making the policy increasingly deterministic\(Cuiet al\.,[2025](https://arxiv.org/html/2608.14610#bib.bib1)\)\. Recent empirical stuides shows that, in RLVR, task\-irrelevant spurious rewards applied with clipping could decrease the LLM’s policy entropy without contributing to its exploitation\.\(Chenet al\.,[2026](https://arxiv.org/html/2608.14610#bib.bib2)\)\.

We hypothesize that TALD is vulnerable to this mechanism\. Newer statutes are often more salient in contemporary legal texts and semantically close to their predecessors\. As a result, applying the newest law can become a high\-probability reasoning trajectory\. If reasoning\-oriented RL reduces exploration over alternative temporal\-applicability paths, it may strengthen this trajectory even when it is legally incorrect\.

##### Hypothesis 5\.

Without hints, explicit reasoning could hurt TALD when version\-related reasoning collapses toward a narrow newest\-law trajectory\.

If this hypothesis holds, interventions that recover TALD performance should be accompanied by higher entropy in version\-related reasoning spans, reflecting reopened exploration over temporal\-law alternatives\.

### 6\.2Measuring TALD Policy Entropy

We use token\-level policy entropy in the model’s CoT as a behavioral proxy for reasoning exploration\. Since TALD failure specifically concerns law versions, we do not measure entropy over the whole response\. Instead, we focus on*version\-related*reasoning spans, including law names, temporal markers, version descriptions, retroactivity, non\-retroactivity, effective periods, and related legal\-temporal expressions\.

For each responseyiy\_\{i\}, letSiS\_\{i\}denote the token positions belonging to version\-related reasoning spans\. At each token positiontt, we compute:

Hi,t=−∑v∈𝒱p​\(v∣yi,<t,xi\)​log⁡p​\(v∣yi,<t,xi\),H\_\{i,t\}=\-\\sum\_\{v\\in\\mathcal\{V\}\}p\(v\\mid y\_\{i,<t\},x\_\{i\}\)\\log p\(v\\mid y\_\{i,<t\},x\_\{i\}\),\(2\)wherexix\_\{i\}is the case fact and𝒱\\mathcal\{V\}is the vocabulary\. The version\-related entropy is:

Hiv​e​r=1\|Si\|​∑t∈SiHi,t\.H\_\{i\}^\{ver\}=\\frac\{1\}\{\|S\_\{i\}\|\}\\sum\_\{t\\in S\_\{i\}\}H\_\{i,t\}\.\(3\)
We report the averageHv​e​rH^\{ver\}over previous\-version cases\.

#### 6\.2\.1Probe Setup

We basically adopt the same evaluation setup as Section\.[5\.2\.2](https://arxiv.org/html/2608.14610#S5.SS2.SSS2), report the TALD policy entropyHiv​e​rH\_\{i\}^\{ver\}together with model performance represented by sensitive\-law recall TALD accuracy on the Old\-Version Split of dataset\.

#### 6\.2\.2Results and Analysis

Table 6:TALD accuracy and version\-relevant policy entropyHiv​e​rH\_\{i\}^\{ver\}on the old\-version split across hint strength\. Each cell reports the mean and standard error over three random seeds\.Table[6](https://arxiv.org/html/2608.14610#S6.T6)reports TALD accuracy and version\-related policy entropy on the Old\-Version Split\. We interpretHv​e​rH^\{ver\}only within the same model family and intervention axis, since absolute entropy values are not directly comparable across tokenizers, probability calibration, and decoding implementations\.

##### Hint perturbations increase version\-related exploration when TALD improves\.

Across Qwen3 and LegalOne, stronger temporal\-law hints generally increase both TALD accuracy andHv​e​rH^\{ver\}\. Qwen3\-30B improves from0\.0200\.020accuracy and0\.3150\.315entropy under no hint to0\.4500\.450accuracy and0\.4620\.462entropy under strong hint\. Qwen3\-80B improves from0\.050/1\.1480\.050/1\.148to0\.578/1\.2110\.578/1\.211, and LegalOne\-8B improves from0\.093/0\.1470\.093/0\.147to0\.298/0\.1730\.298/0\.173\. Qwen3\-235B shows a saturated pattern: accuracy jumps from0\.1550\.155to0\.5240\.524under weak hint, while entropy increases from0\.2380\.238to0\.2530\.253; stronger hint further increases entropy to0\.2660\.266but does not further improve accuracy\.

These patterns suggest that successful TALD correction is accompanied by reopening version\-related reasoning paths, but larger entropy is not itself the objective\. Once a model has reached a better task\-specific policy region, additional uncertainty around version\-related tokens may not produce further accuracy gains\.

##### Finding 6\.

RQ3 provides behavioral evidence for a policy\-entropy explanation of why explicit reasoning can hurt TALD\. Reasoning\-oriented policies may over\-exploit a salient newest\-law trajectory on old\-version cases\. Interventions that improve TALD are generally accompanied by higher entropy in version\-related CoT spans, suggesting that successful TALD requires task\-directed diversity over temporal\-law alternatives rather than merely more explicit reasoning\.

## 7Related Work

### 7\.1Downstream Tasks Related to TALD

A large body of Legal AI work studies downstream prediction tasks from case facts, including legal judgment prediction, charge prediction, law\-article prediction, and legal citation prediction\. Early LJP benchmarks such as CAIL2018\(Xiaoet al\.,[2018](https://arxiv.org/html/2608.14610#bib.bib23)\)formulate judgment prediction as predicting applicable law articles, charges, and penalties from fact descriptions\. Recent work further improves legal judgment prediction by incorporating legal knowledge, multi\-task learning, retrieval, or LLM\-based citation prediction\. These tasks all require models to identify legally relevant authorities before making downstream predictions\(Feiet al\.,[2023](https://arxiv.org/html/2608.14610#bib.bib24); Hanet al\.,[2026](https://arxiv.org/html/2608.14610#bib.bib25); Liuet al\.,[2025](https://arxiv.org/html/2608.14610#bib.bib21)\)\. TALD differs from these tasks by isolating a more basic prerequisite: selecting the temporally applicable version of the relevant law\. Existing article\-prediction or citation\-prediction benchmarks typically treat legal labels as static, whereas real legal reasoning must distinguish semantically related statute versions with different effective periods\. Our work therefore complements downstream legal prediction tasks by diagnosing whether models can first determine which version of law should govern the case\.

### 7\.2Temporal Legal AI

Recent work has begun to examine temporal dynamics in legal AI\. ChronosLex\(Santoshet al\.,[2024](https://arxiv.org/html/2608.14610#bib.bib26)\)studies temporal generalization in legal multi\-label classification by training models on chronological splits\. LexTempus\(Santosh and Vuong,[2025](https://arxiv.org/html/2608.14610#bib.bib28)\)further models legal\-language evolution through a dynamic mixture\-of\-experts framework\. LexTime\(Baraleet al\.,[2025](https://arxiv.org/html/2608.14610#bib.bib22)\)evaluates LLMs’ ability to order legal events, focusing on temporal relations within legal narratives\. Closest to our motivation, LawShift\(Han and others,[2025](https://arxiv.org/html/2608.14610#bib.bib20)\)evaluates legal judgment prediction under statutory revisions and finds that existing models struggle to adapt their judgments when underlying laws change\. Our work differs in bothtaskanddiagnosis\. Rather than studying temporal distribution shift, event ordering, or downstream judgment robustness under synthetic legal amendments, we directly evaluate whether LLMs can identify the temporally applicable statutory version in real judgment\-derived cases\. We further diagnose why models fail by separating version\-centric bias, legal knowledge, general reasoning, task\-specific policy alignment, and version\-related reasoning entropy\.

### 7\.3LLM Reasoning

Reasoning LLMs are achieving state\-of\-the\-art results across a broad range of tasks that require high reasoning ability\. These models are post\-trained with reinforcement learning \(RL\) on verifiable tasks to produce an explicit chain\-of\-thought \(CoT\) before giving a final answer\(Guoet al\.,[2025](https://arxiv.org/html/2608.14610#bib.bib7); Xuet al\.,[2025](https://arxiv.org/html/2608.14610#bib.bib6)\)\. However, tasks of reasoning\-oriented RL training are predominantly limited to mathematics, formal logic, and competitive programming, which are structurally remote from many other domains that could be represented by legal reasoning, which requires navigating long natural\-language texts, domain\-specific principles, and context\-sensitive expert judgment\(Maet al\.,[2026](https://arxiv.org/html/2608.14610#bib.bib5); Chenget al\.,[2026](https://arxiv.org/html/2608.14610#bib.bib4)\)\. Whether reasoning capabilities acquired from such training can generalize to legal reasoning is, therefore, an open and practically important question\. TALD offers a useful testbed for studying this question: unlike many legal reasoning tasks whose assessment is subjective or requires costly expert annotation, the version of law applicable to a given case is uniquely determined by statute and fact, making the outcome objectively verifiable and well\-suited for probing model reasoning quality at scale\.

## 8Conclusion

We introduced TALD, a temporally grounded legal reasoning task that requires models to identify the legally applicable version of statutory law\. Using a benchmark constructed from Chinese civil judgments, we showed that advanced LLMs exhibit a strong newest\-law bias: they perform much better when the newest version applies, but fail severely when previous versions should govern the case\. Fault\-type analysis further shows that these failures are usually version\-centric rather than unrelated legal citations\.

Our diagnostic probes suggest that the failure is not primarily caused by missing statutory knowledge or ignorance of temporal\-effect rules\. Instead, stronger general reasoning does not naturally solve TALD and can even worsen old\-version performance, indicating a misaligned task\-specific reasoning policy\. Finally, our entropy analysis provides behavioral evidence that explicit reasoning may over\-exploit a narrow newest\-law trajectory, while successful correction requires reopening exploration over temporal\-law alternatives\. These findings highlight the need for future legal LLMs to align reasoning not only with general problem\-solving ability, but also with domain\-specific applicability policies\.

##### Limitations\.

First, the scope of our dataset is limited to Chinese civil virdicts, where our annotators can provide reliable legal expertise; although this setting contains rich statutory revisions and well represents the Civil Law Systems, our results may not directly generalize to other legal systems\. Second, we study only the temporal axis of law version\. Other applicability axes, such as jurisdictional version, likewise require selecting one provision among related candidates, and whether reasoning models exhibit analogous default\-policy biases along them could be a meanful next probe\. Third, our mechanistic interpretation is behavioral rather than causal\. Our observation of co\-variation between policy entropy and TALD performance could only be based on open\-weight models, and we were not able to obain observation with intervening on the training objective\. Moving from behavioral evidence to a causal account requires direct intervention on the RL objective itself\.

## 9Ethics Statement

This work evaluates LLM behavior in legal reasoning tasks and should not be used as legal advice\. Incorrect law\-version determination can have serious consequences in real legal settings\. We therefore emphasize that LLM outputs must be verified by qualified legal professionals\. Dataset construction should follow applicable privacy and data\-use rules for judgment documents, including anonymization and removal of personally identifiable information where required\.

## References

- C\. Barale, L\. Barrett, V\. S\. Bajaj, and M\. Rovatsos \(2025\)LexTime: a benchmark for temporal ordering of legal events\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 5220–5236\.Cited by:[§1](https://arxiv.org/html/2608.14610#S1.p3.1),[§7\.2](https://arxiv.org/html/2608.14610#S7.SS2.p1.1)\.
- J\. Bowen \(2005\)Retroactivity and the common law\.Hart Publishing\.Cited by:[§1](https://arxiv.org/html/2608.14610#S1.p2.1)\.
- P\. Chen, X\. Li, Z\. Li, W\. Yin, X\. Chen, and T\. Lin \(2026\)Exploration vs exploitation: rethinking rlvr through clipping, entropy, and spurious reward\.InInternational Conference on Learning Representations,Cited by:[§6\.1](https://arxiv.org/html/2608.14610#S6.SS1.p1.1)\.
- D\. Cheng, S\. Huang, X\. Zhu, B\. Dai, W\. X\. Zhao, Z\. Zhang, and F\. Wei \(2025\)Reasoning with exploration: an entropy perspective\.arXiv preprint arXiv:2506\.14758\.Note:Accepted to AAAI 2026Cited by:[§6\.1](https://arxiv.org/html/2608.14610#S6.SS1.p1.1)\.
- J\. Z\. Cheng, S\. Hao, T\. Liu, F\. Zhou, Y\. Xie, F\. Yao, Y\. Bian, N\. Dey, Y\. Zhuang, Y\. Zha,et al\.\(2026\)Revisiting reinforcement learning for llm reasoning from a cross\-domain perspective\.Advances in Neural Information Processing Systems38\.Cited by:[§7\.3](https://arxiv.org/html/2608.14610#S7.SS3.p1.1)\.
- G\. Cui, Y\. Zhang, J\. Chen, L\. Yuan, Z\. Wang, Y\. Zuo, H\. Li, Y\. Fan, H\. Chen, W\. Chen, Z\. Liu, H\. Peng, L\. Bai, W\. Ouyang, Y\. Cheng, B\. Zhou, and N\. Ding \(2025\)The entropy mechanism of reinforcement learning for reasoning language models\.arXiv preprint arXiv:2505\.22617\.Cited by:[§6\.1](https://arxiv.org/html/2608.14610#S6.SS1.p1.1)\.
- J\. Cui, Z\. Li, Y\. Yan, B\. Chen, and L\. Yuan \(2023\)ChatLaw: open\-source legal large language model with integrated external knowledge bases\.arXiv preprint arXiv:2306\.16092\.Cited by:[§1](https://arxiv.org/html/2608.14610#S1.p1.1)\.
- Z\. Fei, X\. Shen, D\. Zhu, F\. Zhou, Z\. Han, S\. Zhang, K\. Chen, Z\. Shen, and J\. Ge \(2023\)LawBench: benchmarking legal knowledge of large language models\.arXiv preprint arXiv:2309\.16289\.Cited by:[§7\.1](https://arxiv.org/html/2608.14610#S7.SS1.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§7\.3](https://arxiv.org/html/2608.14610#S7.SS3.p1.1)\.
- B\. Hanet al\.\(2025\)LawShift: evaluating temporal generalization of legal judgment prediction\.arXiv preprint\.Cited by:[§1](https://arxiv.org/html/2608.14610#S1.p3.1),[§7\.2](https://arxiv.org/html/2608.14610#S7.SS2.p1.1)\.
- J\. Han, P\. Burgess, and E\. Shareghi \(2026\)Legal citation prediction with llms: a comparative evaluation of instruction tuning, retrieval, and jurisdiction\-specific pre\-training on the auslaw citation benchmark\.Artificial Intelligence and Law\.External Links:[Document](https://dx.doi.org/10.1007/s10506-026-09506-9)Cited by:[§7\.1](https://arxiv.org/html/2608.14610#S7.SS1.p1.1)\.
- D\. Hendrycks, C\. Burns, A\. Chen, and S\. Ball \(2021\)CUAD: an expert\-annotated nlp dataset for legal contract review\.External Links:2103\.06268,[Link](https://arxiv.org/abs/2103.06268)Cited by:[§1](https://arxiv.org/html/2608.14610#S1.p1.1)\.
- Q\. Huang, M\. Tao, C\. Zhang, Z\. An, C\. Jiang, Z\. Chen, Z\. Wu, and Y\. Feng \(2023\)LawyerLLaMA: towards legal large language model\.arXiv preprint arXiv:2305\.15062\.Cited by:[§1](https://arxiv.org/html/2608.14610#S1.p1.1)\.
- Y\. Kryvoi and S\. Matos \(2021\)Non\-retroactivity as a general principle of law\.Utrecht Law Review17\(1\)\.Cited by:[§2\.1](https://arxiv.org/html/2608.14610#S2.SS1.p1.1)\.
- Q\. Li and Q\. Zhang \(2021\)Court opinion generation from case fact description with legal basis\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 14840–14848\.Cited by:[§1](https://arxiv.org/html/2608.14610#S1.p1.1)\.
- J\. Liu, Y\. Tong, H\. Huang, B\. Zheng, Y\. Hu, P\. Wu, C\. Xiao, M\. Onizuka, M\. Yang, and S\. Zheng \(2025\)Legal fact prediction: the missing piece in legal judgment prediction\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 6345–6360\.Cited by:[§1](https://arxiv.org/html/2608.14610#S1.p1.1),[§7\.1](https://arxiv.org/html/2608.14610#S7.SS1.p1.1)\.
- B\. Luo, Y\. Feng, J\. Wang, Z\. Zhang, and D\. Zhao \(2017\)Learning to predict charges for criminal cases with legal basis\.InProceedings of EMNLP,Cited by:[§1](https://arxiv.org/html/2608.14610#S1.p1.1)\.
- X\. Ma, Q\. Liu, D\. Jiang, G\. Zhang, Z\. Ma, and W\. Chen \(2026\)General\-reasoner: advancing llm reasoning across all domains\.Advances in Neural Information Processing Systems38,pp\. 56596–56618\.Cited by:[§7\.3](https://arxiv.org/html/2608.14610#S7.SS3.p1.1)\.
- T\.Y\.S\.S\. Santosh, T\. Vuong, and M\. Grabmair \(2024\)ChronosLex: time\-aware incremental training for temporal generalization of legal classification tasks\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3022–3039\.Cited by:[§7\.2](https://arxiv.org/html/2608.14610#S7.SS2.p1.1)\.
- T\. Santosh and T\. Vuong \(2025\)LexTempus: enhancing temporal generalizability of legal language models through dynamic mixture of experts\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 6608–6624\.Cited by:[§1](https://arxiv.org/html/2608.14610#S1.p3.1),[§7\.2](https://arxiv.org/html/2608.14610#S7.SS2.p1.1)\.
- C\. Xiao, H\. Zhong, Z\. Guo, C\. Tu, Z\. Liu, M\. Sun, Y\. Feng, X\. Han, Z\. Hu, H\. Wang, and J\. Xu \(2018\)CAIL2018: a large\-scale legal dataset for judgment prediction\.arXiv preprint arXiv:1807\.02478\.Cited by:[§7\.1](https://arxiv.org/html/2608.14610#S7.SS1.p1.1)\.
- F\. Xu, Q\. Hao, C\. Shao, Z\. Zong, Y\. Li, J\. Wang, Y\. Zhang, J\. Wang, X\. Lan, J\. Gong,et al\.\(2025\)Toward large reasoning models: a survey of reinforced reasoning with large language models\.Patterns6\(10\)\.Cited by:[§7\.3](https://arxiv.org/html/2608.14610#S7.SS3.p1.1)\.
- J\. Zhang, X\. Wang, F\. Mo, Y\. Zhou, W\. Gao, and K\. Liu \(2025\)Entropy\-based exploration conduction for multi\-step reasoning\.InFindings of the Association for Computational Linguistics: ACL 2025,Vienna, Austria,pp\. 3895–3906\.External Links:[Link](https://aclanthology.org/2025.findings-acl.201/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.201)Cited by:[§6\.1](https://arxiv.org/html/2608.14610#S6.SS1.p1.1)\.

## Appendix ASystem Prompts with Hint Levels

### A\.1Default Prompt \(No Hint\)

> TODO: Insert the exact base prompt used for law article prediction\.

### A\.2Weak Temporal\-Law Hint

> You are a helpful assistant\. Users may ask you legal consultation questions; please respond in a factual, good\-faith manner\. Pay attention to the timing of events in the case and to the law’s temporal applicability / intertemporal effect, especially issues of retroactivity\.

### A\.3Strong Temporal\-Law Hint

> You are a helpful assistant\. Users may ask you legal consultation questions; please respond in a factual, faithful manner\. Pay close attention to the timing of events in the case and to the law’s temporal applicability / intertemporal effect, in particular issues of retroactivity\. In civil law, the basic rule is the principle of non\-retroactivity; therefore, you should carefully examine whether the relevant facts occurred at a time when the prior law should apply\. That said, there are recognized exceptions, such as beneficial retroactivity, retroactive application of newly introduced provisions, and situations in which newly added specific provisions may be invoked as part of the court’s judicial reasoning\.