overfitting

A blog post arguing that AI-generated query optimizations are gaming ClickBench by exploiting its under-specified rules, reviving Goodhart's law in database benchmarking, and proposing fixes like ad-hoc benchmarks, update streams, and benchmarks created after the tested systems.

提出了Harness-Delta Attribution（HDA）评估协议，将agent harness演化带来的性能增益分解为过拟合、测试时计算扩展和可泛化改进三部分，发现11个executor模型和3个框架中过拟合与测试时扩展平均解释了54%和14%的增益，搜索集收益往往无法迁移到held-out任务。

SkillEvoReg introduces a regularization framework to prevent overfitting in the skill evolution of language-model agents, combining dropout, local regularization, and causal validation to maintain performance while controlling skill growth.

This paper introduces Regularized Recursive Self-Improvement (RRSI) for AI agent harnesses, which applies regularization to prevent overfitting during recursive evolution, demonstrating performance gains on multiple benchmarks.

This study validates activation-steering claims for sycophancy in language models, finding no usable linear capitulation direction in two small LLMs and highlighting measurement hazards that underestimate sycophancy.

The article explains why machine learning research agents avoid overfitting during iterative benchmark evaluations, attributing the phenomenon to compression and the role of few-token fits in generalization.

The author argues that viral 'demo-benchmarks' like recreating Minecraft or generating SVG pelicans are easily overfit and measure marketing preparation rather than true AI capability, urging the community to rely on dynamic or private evaluations instead of static public tests.

The paper explains how removing market beta and factor exposure reveals true trading signals, emphasizing rigorous stress testing to avoid overfitting and false confidence in backtests.

Cactus Compute demonstrates that fine-tuning their Needle 2 model on specific tasks can outperform DeepSeek v4 Flash, emphasizing the importance of avoiding benchmark overfitting and providing tools for user customization.

The article highlights how LLMs make it easy to game performance benchmarks, leading to deceptive software optimization claims, as shown with a regex engine that overfits to benchmarks.

The article questions whether theoretical principles still guide machine learning practices, highlighting how many once-standard theories have been challenged by empirical evidence.

AuroSFT introduces a parameter-efficient adapter-based framework for multi-task supervised fine-tuning that rolls back adapter checkpoints at task-wise peaks instead of full-model checkpoints, achieving higher average accuracy than mSFT.

Explores why AI agents that perform perfectly on test cases often fail in real-world conversations, highlighting issues like distribution shift and overfitting.

The author argues that the commonly recommended learning rate of 2e-4 for QLoRA fine-tuning is too high for datasets under 10k samples, leading to overfitting and poor evaluation, and suggests using a lower learning rate like 1e-4.

This paper re-evaluates the methodology of automatic harness evolution for LLM agents, highlighting that its gains may stem from additional test-time search rather than improved harness design, and that evaluation on the same benchmark risks overfitting. Experiments show that harness evolution does not consistently outperform simpler test-time scaling methods.

This paper challenges the common assumption that domain experts for model merging should be trained to their optimal validation loss, showing that the optimal training duration depends strongly on the merging method. Simple averaging degrades with overfitting while sparsification-based methods benefit from training past the optimum, suggesting that training duration and merging method should be chosen jointly.

This paper analyzes the early training collapse phenomenon in deep neural models for click-through rate prediction and proposes mitigation strategies such as sparse feature removal and value filtering, demonstrating improvements on large-scale industrial datasets.

Explores whether small language models (SLMs) can achieve lossless data compression by deliberately overfitting on training data, rethinking the typical aversion to overfitting in machine learning.

This paper explores using ReLU activation statistics as memorization indicators to detect overfitting during fine-tuning of sEMG decoders with limited calibration data, without requiring a validation set.

A developer overfits a small 900KB transformer model to compress a 100MB CSV file down to 7MB, demonstrating a novel approach to data compression using overfitted neural networks.