overfitting

The article explains why machine learning research agents avoid overfitting during iterative benchmark evaluations, attributing the phenomenon to compression and the role of few-token fits in generalization.

The author argues that viral 'demo-benchmarks' like recreating Minecraft or generating SVG pelicans are easily overfit and measure marketing preparation rather than true AI capability, urging the community to rely on dynamic or private evaluations instead of static public tests.

The paper explains how removing market beta and factor exposure reveals true trading signals, emphasizing rigorous stress testing to avoid overfitting and false confidence in backtests.

Cactus Compute demonstrates that fine-tuning their Needle 2 model on specific tasks can outperform DeepSeek v4 Flash, emphasizing the importance of avoiding benchmark overfitting and providing tools for user customization.

The article highlights how LLMs make it easy to game performance benchmarks, leading to deceptive software optimization claims, as shown with a regex engine that overfits to benchmarks.

The article questions whether theoretical principles still guide machine learning practices, highlighting how many once-standard theories have been challenged by empirical evidence.

AuroSFT introduces a parameter-efficient adapter-based framework for multi-task supervised fine-tuning that rolls back adapter checkpoints at task-wise peaks instead of full-model checkpoints, achieving higher average accuracy than mSFT.

Explores why AI agents that perform perfectly on test cases often fail in real-world conversations, highlighting issues like distribution shift and overfitting.

The author argues that the commonly recommended learning rate of 2e-4 for QLoRA fine-tuning is too high for datasets under 10k samples, leading to overfitting and poor evaluation, and suggests using a lower learning rate like 1e-4.

This paper re-evaluates the methodology of automatic harness evolution for LLM agents, highlighting that its gains may stem from additional test-time search rather than improved harness design, and that evaluation on the same benchmark risks overfitting. Experiments show that harness evolution does not consistently outperform simpler test-time scaling methods.

This paper challenges the common assumption that domain experts for model merging should be trained to their optimal validation loss, showing that the optimal training duration depends strongly on the merging method. Simple averaging degrades with overfitting while sparsification-based methods benefit from training past the optimum, suggesting that training duration and merging method should be chosen jointly.

This paper analyzes the early training collapse phenomenon in deep neural models for click-through rate prediction and proposes mitigation strategies such as sparse feature removal and value filtering, demonstrating improvements on large-scale industrial datasets.

Explores whether small language models (SLMs) can achieve lossless data compression by deliberately overfitting on training data, rethinking the typical aversion to overfitting in machine learning.

This paper explores using ReLU activation statistics as memorization indicators to detect overfitting during fine-tuning of sEMG decoders with limited calibration data, without requiring a validation set.

A developer overfits a small 900KB transformer model to compress a 100MB CSV file down to 7MB, demonstrating a novel approach to data compression using overfitted neural networks.

This paper investigates training-time data augmentation techniques to mitigate overfitting in autoregressive language model pretraining under data-constrained, compute-abundant regimes, finding that combining token-level noise, sequence permutations, and target offset prediction improves validation loss.

This paper introduces the log-alignment ratio (LAR), a training-time metric that measures parameter-activation alignment and predicts generalization by capturing the spread of weight and activation spectra. Experiments on grokking and a 3B-parameter language model show LAR tracks the transition from memorization to generalization and flags overfitting without held-out data.

Proposes R2R2, a regularization method for self-predictive learning in reinforcement learning to mitigate overfitting under high update-to-data ratios, achieving significant improvements on continuous control tasks.

A user found that reducing the learning rate from 2e-4 to 1e-4 significantly improved QLoRA fine-tuning of Llama 3.1 8B on a small dataset (8k samples), preventing overfitting and leading to better evaluation results.

This paper studies the trade-off between scarce target data and abundant generic data in mixture pretraining, finding that repetition is a key driver of performance and that mixture training tolerates 15-20 repetitions of target data. It introduces a repetition-aware scaling law to optimize mixture configurations under data constraints.