Reasoners or Translators? Contamination-aware Evaluation and Neuro-Symbolic Robustness in Tax Law
Summary
This paper empirically studies LLMs' legal reasoning in tax law, showing that data contamination inflates performance and that neuro-symbolic hybrid systems offer more reliable and robust generalization than monolithic LLMs.
View Cached Full Text
Cached at: 05/18/26, 06:35 AM
# Reasoners or Translators? Contamination-aware Evaluation and Neuro-Symbolic Robustness in Tax Law Source: [https://arxiv.org/abs/2605.16052](https://arxiv.org/abs/2605.16052) [View PDF](https://arxiv.org/pdf/2605.16052) > Abstract:Recent advances in large language models \(LLMs\) have significantly enhanced automated legal reasoning\. Yet, it remains unclear whether their performance reflects genuine legal reasoning ability or artifacts of data contamination\. We present a comprehensive empirical study of tax law reasoning approaches and implement a contamination detection protocol to rigorously assess LLM reliability\. We show that performance can be inflated by contamination\. Building on this analysis, we conduct a systematic evaluation, comparing monolithic LLMs with hybrid systems that translate statutory text into formal representations and delegate inference to symbolic solvers\. We build a novel test suite designed to probe generalization to unseen documents via case and rule variations\. Our findings indicate that legal reasoning is inherently compositional and that neuro\-symbolic frameworks offer a more reliable and robust foundation for legal AI, as well as improved generalization to unobserved situations\. ## Submission history From: Parisa Kordjamshidi \[[view email](https://arxiv.org/show-email/a2c105b0/2605.16052)\] **\[v1\]**Fri, 15 May 2026 15:20:16 UTC \(129 KB\)
Similar Articles
Reasoning Before Translation: Enhancing Legal Machine Translation with Structured Reasoning
This paper compares various approaches to enhance neural machine translation in the legal domain, including supervised fine-tuning and reinforcement learning with verifiable rewards. It focuses on Swiss legal system translations and shows that small language models can be greatly enhanced, with RL surpassing SFT performance.
Which Changes Matter? Towards Trustworthy Legal AI via Relevance-Sensitive Evaluation and Solver-Grounded Reasoning
This paper introduces a relevance-sensitive evaluation suite for legal AI, demonstrating that LLMs are overly sensitive to legally irrelevant perturbations, and proposes LexGuard, an adversarial multi-agent framework using formal reasoning to improve legal reasoning reliability.
Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms
This paper investigates how large language models perform arithmetic operations by analyzing internal mechanisms through early decoding, revealing that proficient models exhibit a clear division of labor between attention and MLP modules in reasoning tasks.
Hidden Language Consistency Phenomena in Reasoning LLMs
This paper studies multilingual reasoning models and reveals that language consistency in outputs can degrade or collapse with task difficulty, especially for less represented languages. It argues that evaluating multilingual capability requires jointly considering accuracy, language consistency, and task difficulty.
When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitions
This paper investigates when chain-of-thought reasoning is beneficial for LLMs, showing that early-stage entropy dynamics reliably indicate reasoning utility, and introduces EDRM, a lightweight, training-free framework that adaptively selects inference strategies to achieve significant token savings while maintaining or improving accuracy.