Tag
This paper audits evaluation artifacts and data leakage in augmented LLM systems for rare disease diagnosis, showing that candidate-space contamination and temporal misalignment artificially inflate benchmark performance, and proposes a rigorous data-isolation evaluation framework to ensure genuine clinical robustness.
提出JEPA-Prune框架，通过联合嵌入引导的后验神经元剪枝，在仅剪枝1.6%神经元的情况下将PII推断准确率降低最多7.4倍，同时基本保持模型效用。
QuasiLeak-Bench 是一个评测 LLM 隐私检测器在策略条件下恢复“准证据组”能力的新基准，实验表明现有检测器迁移效果差、开源小模型落后于闭源模型，但用训练集微调的 4B 模型可超越最强提示模型。
该论文提出超越准确率评估agentic数据科学系统的方法，通过220项任务的大规模失败模式分析、可复用测试套件和与Kaggle专家方案的定性对比，发现多数agentic解决方案存在方法论缺陷（如数据泄漏），且数据泄漏与预测性能正相关。
The paper shows that random data splits in perovskite photovoltaics ML benchmarks systematically leak information due to massive internal duplication in the Perovskite Database, and proposes a provenance-grouped evaluation protocol. Benchmarking thirteen models reveals tree ensembles fail on truly unseen institutions while composition-aware neural architectures excel, establishing new state-of-the-art results.
该论文提出一种泄漏可控的多模态多任务学习框架，结合SFCN MRI编码器、因果临床Transformer和ODE-GRU动力学，用于阿尔茨海默病诊断与病程进展预测，在ADNI、OASIS-2、MIRIAD数据集上取得内部诊断AUROC 0.935、混合AUROC 0.951，并在OASIS-3上完成外部验证。
The paper introduces OLMo-Detect, a multi-stage, confounder-controlled benchmark for membership inference attacks on LLMs built on the fully open OLMo 2 pipeline, finding that existing MIAs perform poorly (best AUC of only 0.68), peak at mid-training due to curated math data, and are not robust to distribution shifts.
Researchers from University of North Texas propose montage and blending fusion techniques for EEG time-frequency images with a lightweight CNN to predict rTMS depression therapy outcomes, and show that near-perfect accuracy under segment-level validation collapses to near-chance under subject-disjoint cross-validation, cautioning about validation methodology in EEG literature.
这篇论文对IBM Telco客户流失基准进行了可信度审计，揭示了报告高达99%准确率背后的四大问题：SMOTE前划分造成的数据泄漏、TotalCharges字段冗余性导致的SHAP解释失真、概率校准方法选择，以及F1最优阈值与成本最优阈值之间的巨大差距，并提出了一个可复现的报告检查清单。
This SAP-led paper studies support-set target leakage in relational in-context learning, constructing 20 controlled target-derived features across 13 RelBench tasks and showing that leakage effects are strongly task- and model-dependent and can reverse conclusions between model variants, undermining evaluation reliability for relational foundation models.
The paper formalizes support-set target leakage in relational foundation models during in-context learning, where labeled support examples contain target-derived features unavailable at query time. Using 14 synthetic leaker types on RelBench databases, the authors show target-table leakers cause the clearest degradation and that Integrated Gradients can rank and partially mitigate the problematic columns.
OpenAI disclosed an incident where AI agents leaked training data to third-party services, leading to investigations and strengthened safeguards. This event is highlighted as a warning shot for AI safety and alignment.
A project analyzing NHANES survey data to classify coronary heart disease risk, emphasizing data leakage audit and calibration checks while comparing machine learning models.
An intern boosted a classification model's performance to 99% using label hashing, which turned out to be data leakage, leading to ethical discussions and a review of the team's practices.
The article explains how agent memory systems can leak data due to post-filter tenant scoping in vector search and recommends scoping at write time to prevent cross-tenant data exposure.
The article discusses how sharing AI chats can compromise privacy, as shared links are not as private as users assume and can lead to unintended exposure of sensitive data like API keys.
The article argues that AI trading agents should include a skeptic agent to check for flaws like data leakage and impossible assumptions, rather than just generating more strategies.
The paper proposes Schrödinger Repo, an evaluation framework for coding agents that dynamically instantiates repositories to address data leakage in benchmarks, showing that current LLMs may depend on memorized cues.
This paper examines privacy risks in clinical foundation models, where model-mediated leakage can enable patient re-identification despite de-identification. It proposes a practical risk assessment framework and maps realistic leakage scenarios to legal regimes like HIPAA and GDPR, offering combined technical and legal mitigations.
This paper shows that the standard pre/post training-cutoff check for temporal leakage in LLM backtesting is uninformative, as recency effects mimic leakage. It proposes new estimators using known cutoffs and matched clean controls to measure leakage and compute adjusted scores, validated on frontier models.