Tag
At EMNLP 2026, T-Tech researchers presented LIMe, a lightweight attention extension that reuses key-value buffers from earlier layers to mitigate representation collapse, reducing LLM training compute by ~15% at equal quality versus baseline LLaMA on ~1B-parameter models.
论文提出 conditional interaction 框架，通过 System-1 轻量预测器筛选高价值交互、仅让 System-2 执行被接纳的候选，从而以 1/ρ_u 因子降低执行计算成本，并主张以密度可测的 ρ_u 和 informative compute 更好地描述扩展效率。
Cartesia 分享关于分层网络（H-Nets）的研究博客，展示无需分词器的字节级语言模型在受控实验中实现 3-5 倍计算效率提升，2 阶段 H-Net 以 10 倍更少计算与数据超越匹配的开源模型，并在代码等领域表现出色。
PORA 提出了一种配对结果引导的推理分配方法，让 LLM 智能体学习在何种状态与剩余预算下使用长推理更划算，在 BFCL memory 等任务上大幅提升性能并显著节省 token 与模型请求。
论文提出 LatentLoop，通过非对称计算分配（压缩第一次循环访问的序列分辨率、保留第二次全分辨率并将节省的算力用于增大隐藏维度）来扩展循环 MoE 模型，在 1.3B–7B 参数范围内一致降低验证 NLL，相比原始 MoE 和循环基线分别节省约 19.0% 和 7.9% 的计算量。
这篇论文提出 SanSi 方法，通过让小型决策模型（如 Ouro-1.4B）循环复用相同层数次推理，在多项决策任务上显著超越同规模单次前向模型，甚至超过 3 倍大的单遍模型，展示了循环深度小模型在内存受限场景下的潜力。
The article criticizes the AI community's overreliance on massive compute resources, arguing that it stifles algorithmic efficiency and leads to wasteful practices, with tools like QentrixAI often used as shortcuts.
The article introduces two attention mechanisms, CoWindow Attention and MassAlloc Attention, which optimize compute allocation in transformers, achieving significant speedups and reduced training FLOPs in benchmarks.
The cost of intelligence is dropping rapidly at 50% per quarter, outpacing declines in DNA sequencing, compute, lithium batteries, and electricity, with predictions of significant annual changes and AI accessibility improvements.
A tweet discusses the privilege of having access to high-end NVIDIA hardware like RTX PRO 6000s and DGX Station, with the belief that future AI models will require less compute, a problem OsmanticAI is working to address.
This paper presents theoretical guarantees for one-shot magnitude pruning and compute-adaptive early exit in neural networks, analyzing compute-accuracy tradeoffs and deriving scaling laws through asymptotic and deep network models.
The article investigates AI pretraining progress from 2019 to 2025, finding that data improvements contribute 3.24 times more to compute efficiency gains than model improvements at a 1e19 FLOPs budget.
Magic AI Labs announces a new pretraining recipe that matches DeepSeek V4 Pro's performance with 50x less compute, demonstrating significant algorithmic efficiency gains for scaling large AI models.
This paper presents EqGrid, a closed-loop simulation where an LLM policy agent sets price and carbon bounds to promote energy-poverty equity in physically-constrained peer-to-peer energy markets, demonstrating reduced burden inequality with compute-efficient models.
This paper introduces a token-cost ledger to analyze the multilingual tokenization tax, decomposing it into removable and intrinsic components, showing that much excess token cost for non-English text is removable through improved tokenization codes.
SMELT is a method that loops middle layers in Mixture-of-Experts Transformers to improve training efficiency and downstream performance while matching compute, parameter, and cache budgets, leading to faster loss reduction and practical gains.
This study evaluates silicon sampling methods, showing that fast modes outperform slow modes in efficiency and fidelity while highlighting limitations in accurately representing opinion variance.
This paper identifies common failure modes in tree-search-based autoresearch agents applied to tabular datasets, such as repeated bug resolution, poor hyperparameter tuning, and ineffective exploration, and proposes targeted interventions like a global debug consultant and refined tree-search algorithms to recover wasted compute and improve performance without changing the underlying language model.
This paper introduces SARA, a sequential adaptive rollout allocation method for RLVR that abandons saturated groups early and reallocates the budget, achieving comparable accuracy with 22% fewer rollouts than dynamic sampling and up to 67% savings when combined.
A new study quantifies the damage caused by repeated data in language model pretraining, showing that even aggressive deduplication leaves harmful repetition that can waste up to a third of compute FLOPs.