Tag
This paper presents theoretical guarantees for one-shot magnitude pruning and compute-adaptive early exit in neural networks, analyzing compute-accuracy tradeoffs and deriving scaling laws through asymptotic and deep network models.
The article investigates AI pretraining progress from 2019 to 2025, finding that data improvements contribute 3.24 times more to compute efficiency gains than model improvements at a 1e19 FLOPs budget.
Magic AI Labs announces a new pretraining recipe that matches DeepSeek V4 Pro's performance with 50x less compute, demonstrating significant algorithmic efficiency gains for scaling large AI models.
This paper presents EqGrid, a closed-loop simulation where an LLM policy agent sets price and carbon bounds to promote energy-poverty equity in physically-constrained peer-to-peer energy markets, demonstrating reduced burden inequality with compute-efficient models.
This paper introduces a token-cost ledger to analyze the multilingual tokenization tax, decomposing it into removable and intrinsic components, showing that much excess token cost for non-English text is removable through improved tokenization codes.
SMELT is a method that loops middle layers in Mixture-of-Experts Transformers to improve training efficiency and downstream performance while matching compute, parameter, and cache budgets, leading to faster loss reduction and practical gains.
This study evaluates silicon sampling methods, showing that fast modes outperform slow modes in efficiency and fidelity while highlighting limitations in accurately representing opinion variance.
This paper identifies common failure modes in tree-search-based autoresearch agents applied to tabular datasets, such as repeated bug resolution, poor hyperparameter tuning, and ineffective exploration, and proposes targeted interventions like a global debug consultant and refined tree-search algorithms to recover wasted compute and improve performance without changing the underlying language model.
This paper introduces SARA, a sequential adaptive rollout allocation method for RLVR that abandons saturated groups early and reallocates the budget, achieving comparable accuracy with 22% fewer rollouts than dynamic sampling and up to 67% savings when combined.
A new study quantifies the damage caused by repeated data in language model pretraining, showing that even aggressive deduplication leaves harmful repetition that can waste up to a third of compute FLOPs.
This paper introduces CHERRY, a set of techniques for compute-efficient language models including selective token supervision, depth compression via recurrent unrolling, and a mixture of compressed experts, achieving significant efficiency gains on a Korean foundation model.
This paper systematically studies the damage caused by exact document repetition during language model pretraining, showing that repeating a moderately sized subset a moderate number of times maximally harms performance, and that repetition can waste up to 33% of compute (as measured by compute-equivalent loss).
ZeroGPU is a compute efficient layer designed for AI inference, aiming to optimize GPU usage and reduce costs.
LayerRoute is a lightweight adapter that selectively skips transformer blocks during inference based on input type, achieving compute savings while maintaining or improving model quality through gated routing and LoRA adaptation. It achieves a 12.91% skip differential on agentic language models.
LVSA introduces a training-free sparse attention mechanism for video diffusion models, reducing compute up to 3.17x while enabling generation beyond training horizons without quality loss.
This paper introduces LBW-Guard, a bounded autonomous training control governance layer that operates above the AdamW optimizer to monitor telemetry and apply bounded control during training, demonstrating improved perplexity and training speed under stress conditions.
Discusses the potential for AI-rendered video to be far more compute-efficient than traditional rendering, using Big Hero 6's millions of render hours as a benchmark.
A method that dynamically allocates compute budget to hard problems using Qwen-35B-A3B achieves performance near GPT-5.4-xHigh on the HLE benchmark.
A new optimization technique for open-source RL training engines introduces prompt caching during training, achieving up to 7.5x speedup on long-prompt, short-response workloads by reducing redundant compute.
Foundational empirical study demonstrating power-law scaling relationships between language model performance and model size, dataset size, and compute budget, with implications for optimal training allocation and sample efficiency.