Tag
Introduces Uncheatable Eval, a dynamic benchmark using compression rates to evaluate language models and mitigate data contamination.
This paper measures how AI agents contaminate stores they write to, identifying a threshold for error propagation and validating findings with synthetic and real-world Wikidata data.
A tweet discusses concerns about an AI benchmark being included in training data, with Simon Willison sharing a related movie trivia fact.
This paper demonstrates that benchmark contamination inflates absolute scores for large language models but rarely reorders leaderboard rankings, using a paraphrase-controlled measure to show contamination is largely uniform across public models.
This paper proposes CalibDCD, a calibration framework for feature-based LLM data contamination detection that mitigates feature shifts caused by post-training, improving detection performance by up to 7.0% AUC and 15.0% TPR@5%FPR.
Introduces Cultivar, a contrastive localized translation benchmark for detecting data contamination and evaluating localization robustness in multilingual translation models. It benchmarks 32 open-weight models and finds that MT-specialised models are less robust, with potential overfitting to FLORES.
This paper evaluates LLMs on understanding and generating novel Chinese xiehouyu riddles, finding that while models like Gemini 3.1 Pro excel at comprehension due to memorization, they fall short in creative generation compared to humans, highlighting issues of data contamination and limits of reasoning.
CursorBench reveals that Grok 4.5's high scores were partly due to unintentional inclusion of an earlier snapshot of the Cursor codebase in its training data. The data has been removed for future models.
A detailed explanation of why training on benchmarks, evals, or test sets is a cardinal sin in ML, corrupting the ability to measure generalization. The article emphasizes the importance of clean evaluation protocols and warns against benchmaxxing.
This paper proposes recall-based prompting strategies (Self-Recall and Question-Recall) to improve LLM knowledge cutoff adherence, outperforming existing methods on counterfactual questions and introducing a Multi-cutoff Historical Event Benchmark (MHEB) for robustness evaluation.
LaRA is a layer-wise representation analysis framework that detects data contamination in RL post-trained LLMs by measuring geometric deviations across model layers, outperforming output-level baselines.
This paper introduces TSFMAudit, the first method for auditing pretraining data contamination in time series foundation models, using probe adaptation dynamics to detect unusually efficient fine-tuning that indicates prior exposure.
A unified survey of pretraining data exposure (PDE) in large language models, covering membership inference, data contamination, and security implications, with a review of attack and defense methods.
Proposes Joint Envelope Conformal Selection (JECS), a conformal procedure for multi-model benchmark decontamination that provably controls global contamination rate while maintaining higher power than baselines.
This paper introduces Zero-CoT Probe (ZCP), a black-box detection method that identifies evasive data contamination in LLMs by truncating chain-of-thought reasoning and comparing performance on perturbed datasets, achieving robust detection of both direct and indirect contamination.
This paper investigates LLM-based generative error correction (GER) for low-resource West Frisian ASR, using a contamination-aware evaluation with a private dataset to show that GPT-5.1 reduces errors beyond oracle levels.
This paper empirically studies LLMs' legal reasoning in tax law, showing that data contamination inflates performance and that neuro-symbolic hybrid systems offer more reliable and robust generalization than monolithic LLMs.
Hugging Face announces the addition of private, high-quality datasets from Appen and DataoceanAI to the Open ASR Leaderboard to prevent benchmaxxing and test-set contamination, while maintaining public data for the default average WER calculation.