Tag
Prime Intellect made DeepSeek's MegaMoE kernels trainable, fusing computation and communication in the MoE layer to achieve up to 55% faster MoE training than TorchTitan, with support for BF16, MXFP8, and FP8 on Blackwell and Hopper GPUs.
Soft-MTP 提出一种 KL 梯度恢复技巧，去除多token预测训练中的噪声残差梯度，在无额外开销下改进困惑度、投机解码与推理能力。
该论文提出了一种经验性加法幂律，将长上下文持续预训练（LCCP）的收益与模型规模及短/长上下文token预算关联起来，并引入Constant-Last4协议实现稳定监控。实验表明LCCP的收益在评测范围内无明显饱和迹象，主张不应过早停止长上下文持续预训练。
该论文提出 FlashNext-RL——一个针对 Qwen3.8-Flash-Next 的模型专用长时程 agentic RL 训练优化框架，基于 prime-rl 与 vLLM，在单个 8×B200 节点上将 BF16 全模型训练吞吐从约 395 tokens/s/GPU 提升至 4,438 tokens/s/GPU（131K token 上下文），并支持实时 SWE RL 训练运行。
该论文提出ΣΔ-Sampler优化器，将二值和三值神经网络训练损失映射为高阶Ising模型能量，并通过噪声整形与退火实现高效的边缘设备训练，在CIFAR-10、ImageNet-1k和MatMul-free语言模型上验证了接近全精度的精度，同时将训练内存减少超过50%、速度提升一个数量级。
The article introduces two attention mechanisms, CoWindow Attention and MassAlloc Attention, which optimize compute allocation in transformers, achieving significant speedups and reduced training FLOPs in benchmarks.
This paper systematically analyzes length-based training in speech token language models, showing that short-to-long ordering has no independent benefit when batch composition and token exposure are fixed, and evaluates the impact of loss normalization.
Temperon is a training method that uses Sharpness-aware minimization (SAM) only in the final phase of training to achieve full SAM quality with a third less wall-clock time, validated on vision and language tasks.
This paper introduces HarnessBandit, an online scheduler for training language-model agents across multiple harnesses by jointly optimizing learnability and transferability signals, improving performance on benchmarks like PinchBench and ClawEval.
NanoForecast v0.5 is a 6.5M-parameter time series forecaster that achieves competitive performance against larger models like TimesFM by optimizing the training pipeline, resulting in a 43.8% reduction in MASE. The model and code are released under Apache 2.0.
This paper explores methods to mitigate sleep deprivation effects in the Forward-Forward algorithm, showing accuracy improvements of 2%-62% on MNIST and Fashion-MNIST datasets through techniques like alternative activation and loss optimization.
This paper introduces HybridAL, a method for active learning that dynamically switches between retraining and fine-tuning based on a stabilization signal, improving efficiency and performance in text classification tasks.
The blog post identifies a silent expert death failure mode in ultra-sparse MoE training and introduces LM Loss as Auxiliary Loss (LLAL) to rescue lower-layer experts, improving validation loss and downstream performance.
ERASE introduces a novel training schedule that detaches subgraphs to overlap backward passes with forward work, improving throughput by up to 9.51% in large-scale recommendation systems while preserving model performance.
This paper presents HiFloat4 format and Rollout-ResQ for end-to-end FP4 reinforcement learning post-training of LLMs, achieving a 1.1% accuracy gap to BF16 on Qwen2.5 models.
A detailed guide on designing effective ML experiments, emphasizing starting with a clear research question, developing research taste, and scaling results. Based on the author's experience running ~100 experiments weekly at Poolside.
Anthropic's Mythos system achieved a 52x speedup in optimizing training code compared to a human's 4x speedup over 4-8 hours on the same task, with the caveat that absolute multiples depend heavily on starting code quality. The like-for-like comparison shows roughly 3x–52x improvement across models over the past year.
This paper proposes PAT, an adaptive tensor parallelism method that dynamically reconfigures TP during the generation stage of synchronous RLHF training to mitigate long-tail generation bottlenecks. Evaluations on LLaMA3.1-8B and Qwen3-14B show reductions in generation latency by up to 34.6% and end-to-end iteration latency by up to 27.2%.
A new optimization technique for open-source RL training engines introduces prompt caching during training, achieving up to 7.5x speedup on long-prompt, short-response workloads by reducing redundant compute.
Photoroom's PRX Part 3 demonstrates training a text-to-image model in 24 hours by combining optimized architectural and training techniques including perceptual losses, token routing with TREAD, and the Muon optimizer.