The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation
Summary
This paper introduces Zero-CoT Probe (ZCP), a black-box detection method that identifies evasive data contamination in LLMs by truncating chain-of-thought reasoning and comparing performance on perturbed datasets, achieving robust detection of both direct and indirect contamination.
View Cached Full Text
Cached at: 05/25/26, 06:39 PM
Paper page - The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation
Source: https://huggingface.co/papers/2605.21856
Abstract
A black-box detection method called Zero-CoT Probe is introduced to identify data contamination in large language models by truncating reasoning processes and comparing performance on original and perturbed datasets.
Large language models (LLMs) have demonstrated impressive reasoning abilities across a wide range of tasks, butdata contaminationundermines the objective evaluation of these capabilities. This problem is further exacerbated by malicious model publishers who use evasive, or indirect, contamination strategies, such as paraphrasing benchmark data to evade existing detection methods and artificially boost leaderboard performance. Current approaches struggle to reliably detect such stealthy contamination. In this work, we uncover a critical phenomenon: a model’s generated reasoning steps actively mask its underlyingmemorization. Inspired by this, we propose theZero-CoT Probe(ZCP), a novelblack-box detectionmethod that deliberately truncates the entireChain-of-Thought(CoT) process to expose latent shortcut mappings. To further isolatememorizationfrom the model’s intrinsic problem-solving capabilities, ZCP compares the model’s zero-CoT performance on the original benchmark against anisomorphically perturbed reference dataset. Furthermore, we introduceContamination Confidence, a metric that quantifies both the likelihood and severity of contamination, moving beyond simple binary classifications. Extensive experiments on both previously identified contaminated models and specially fine-tuned contaminated models demonstrate that ZCP robustly detects both direct and evasivedata contamination. The code for ZCP is accessible at https://github.com/Yifan-Lan/zero-cot-probe.
View arXiv pageView PDFGitHub1Add to collection
Get this paper in your agent:
hf papers read 2605\.21856
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.21856 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.21856 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.21856 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
The strange thing about LLM reasoning research: we're now trying to remove the chain-of-thought traces
The article discusses a shift in LLM reasoning research from making reasoning explicit via chain-of-thought to exploring latent reasoning that doesn't require language traces, questioning whether visibility is necessary for effective reasoning.
Hidden Thoughts Are Not Secret: Reasoning Trace Exposure in LLMs
This paper introduces Reasoning Exposure Prompting (REP), a method that uses shadow-model demonstrations in code-like formats to elicit hidden reasoning traces from LLMs, showing that interface-level trace hiding is insufficient to prevent extraction of useful reasoning signals.
Parallel LLM Reasoning for Bias-Resilient, Robust Conceptual Abstraction
This paper proposes a framework for parallel chunk-level processing of long documents with LLMs to reduce cumulative bias and improve evidence traceability, achieving significant reductions in omission errors and unsupported claims.
Decoding Hidden Deception in Reasoning LLMs: Activation Explainers for Deception Auditing
Introduces STATEWITNESS, an activation explainer for auditing deception in reasoning LLMs, achieving significant improvements over existing monitors and providing human-inspectable evidence.
Efficient Reasoning Training Does Not Always Harm CoT Faithfulness and Monitorability
This paper investigates whether training LLMs for shorter, more efficient chain-of-thought reasoning harms the faithfulness and monitorability of that reasoning. It finds faithfulness generally declines due to reduced model consistency, while monitorability remains relatively robust even with much shorter CoT outputs.