efficient-training

Tag

Cards List
#efficient-training

DiffusionRVGE: Reinforcement Learning via Regression-Based Value Gradient Estimation

arXiv cs.AI ↗ · 2026-10-19 Cached

DiffusionRVGE 提出一种基于回归的价值梯度估计方法,通过对中心化奖励与中心化去噪图像进行回归来构建扩散模型的 RL 更新方向,无需奖励导数或穿过采样轨迹的反向传播,在匹配 DiffusionNFT 效果的同时显著降低训练计算成本。

0 favorites 0 likes
#efficient-training

Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation

arXiv cs.AI ↗ · 2026-10-05 Cached

The paper introduces Success-Referenced On-Policy Distillation (SR-OPD), which uses a student's own successful rollouts as references to selectively allocate teacher supervision, achieving comparable reasoning performance on math benchmarks using only 3.46–5.02% of the teacher-input tokens required by vanilla on-policy distillation.

0 favorites 0 likes
#efficient-training

Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models

Hugging Face Daily Papers ↗ · 2026-10-02 Cached

Pivot-SD(EMNLP 2026 Oral)提出一种面向掩码扩散语言模型的高效自蒸馏方法,通过信息增益识别关键的"pivot"去噪步骤并仅对这些步骤进行奖励/惩罚,仅用200个问题在数学与代码任务上以更少算力超越全序列SFT和在线RL基线。

0 favorites 0 likes
#efficient-training

SMAT: Simple and Efficient Merge-Aware Training

Hugging Face Daily Papers ↗ · 2026-09-27 Cached

SMAT introduces a simple and efficient merge-aware training method that jointly optimizes expert loss and expected loss at simulated merged parameters, improving merged model performance with minimal computational overhead.

0 favorites 0 likes
#efficient-training

@willdepue: great paper, don’t buy the width mixing (i think fixed width probably better) and feel like they should be able to get …

X AI KOLs Following ↗ · 2026-08-21 Cached

The paper introduces Matryoshka Language Model Suites, which nests multiple model sizes (500M, 1.5B, 3B) into a single architecture for joint training, allowing smaller models to benefit from distillation with the largest model.

0 favorites 0 likes
#efficient-training

Aurora-80K releases! A modern tiny language model.

Reddit r/LocalLLaMA ↗ · 2026-08-20 Cached

Aurora-80K is a modern tiny language model with 80 thousand parameters, trained on 40M tokens and released on Hugging Face with benchmark results.

0 favorites 0 likes
#efficient-training

Dion3: Full-Stack Orthogonal Updates

arXiv cs.LG ↗ · 2026-08-13 Cached

Dion3 is a revised Muon optimizer that reduces computational and communication overhead via Gram Newton-Schulz, symmetric GEMM kernels, and megabatching, achieving up to 6x faster optimizer steps while matching or improving loss.

0 favorites 0 likes
#efficient-training

LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training

arXiv cs.CL ↗ · 2026-08-13 Cached

LazyTrain is a research system that optimizes LLM training on limited hardware by formulating checkpoint selection, activation placement, recomputation, and CPU-GPU-NVMe communication overlap as a mixed-integer scheduling problem, achieving ~1.24x TFLOPS improvement over matched baselines.

0 favorites 0 likes
#efficient-training

Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts

arXiv cs.LG ↗ · 2026-08-12 Cached

This paper introduces MOSAIC, a framework that jointly optimizes sparse Mixture-of-Experts model architecture and hardware systems for large-scale pretraining, showing that compute-optimal sparsity is not necessarily cluster-optimal when MFU, communication costs, and parallel layouts are considered.

0 favorites 0 likes
#efficient-training

Interpreting Language Model Hidden States at Scale

arXiv cs.AI ↗ · 2026-08-12 Cached

OmniLens is a scalable lens method for interpreting LLM hidden states, using low-rank translators and Subset-KL to reduce parameters and memory, enabling a dense ensemble of 482 lenses on LLaMA-3.3-70B at substantially lower cost.

0 favorites 0 likes
#efficient-training

HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models

arXiv cs.LG ↗ · 2026-07-30 Cached

This paper presents HiFloat4 format and Rollout-ResQ for end-to-end FP4 reinforcement learning post-training of LLMs, achieving a 1.1% accuracy gap to BF16 on Qwen2.5 models.

0 favorites 0 likes
#efficient-training

DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning

arXiv cs.LG ↗ · 2026-07-28 Cached

DomainPilot introduces a domain-level loss-guided two-stage framework for data mixture optimization in LLM fine-tuning, achieving improvements on MMLU-Redux, AIME24, LiveCodeBench v5, and BFCL v3 without increasing data volume or training cost.

0 favorites 0 likes
#efficient-training

Learning Faster without Deeper Networks: A*-Inspired Batch Selection for Efficient CNN Training

arXiv cs.LG ↗ · 2026-07-20 Cached

The paper proposes A*-Inspired Batch Selection (A*-BS), a lightweight strategy that treats mini-batch scheduling as a heuristic search to improve CNN training efficiency. On MedMNIST tasks, a simple CNN with A*-BS outperforms deeper ResNet-18/50 baselines in accuracy and training speed.

0 favorites 0 likes
#efficient-training

@j_golebiowski: How long it takes to build a task-specific local model, start to finish: - 25 hand-written examples (one afternoon) - e…

X AI KOLs Timeline ↗ · 2026-07-15 Cached

Describes a workflow to build a task-specific local model in under a day using 25 hand-written examples, synthetic data expansion, LoRA fine-tuning, and quantization for CPU inference.

0 favorites 0 likes
#efficient-training

Efficient Long-Horizon Learning for Learned Optimization

arXiv cs.LG ↗ · 2026-07-09 Cached

Proposes Efficient Long-horizon Optimization (ELO) learning, a meta-training algorithm that reallocates compute to longer horizons and uses decoupled progressive expert supervision, improving learned optimizers' performance on long-unroll tasks and out-of-distribution generalization. ELO-Celo2 consistently outperforms AdamW and matches Muon on language modeling tasks.

0 favorites 0 likes
#efficient-training

TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training

arXiv cs.AI ↗ · 2026-07-08 Cached

TurnOPD introduces turn-level budgeting for on-policy distillation of long-horizon agents, addressing inefficiencies in vanilla OPD by adaptive rollout-depth and progressive turn-normalized loss budgeting, achieving better accuracy under equal training budgets.

0 favorites 0 likes
#efficient-training

SiamJEPA: On the Role of Siamese Student Encoders in JEPA

Hugging Face Daily Papers ↗ · 2026-07-04 Cached

This paper proposes SiamJEPA, which uses masked Siamese student encoders with an EMA teacher network in JEPA models, showing improved representation separability and training efficiency compared to single-encoder variants and MAE.

0 favorites 0 likes
#efficient-training

Token Geometry

arXiv cs.LG ↗ · 2026-07-03 Cached

The paper introduces Ember, a lightweight optimizer for embedding and LM-head matrices that exploits gradient geometry to improve efficiency and performance across supervised finetuning, RL, and pretraining, while using far less optimizer state than Adam.

0 favorites 0 likes
#efficient-training

DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training

arXiv cs.LG ↗ · 2026-06-30 Cached

Introduces Duplicated Latent Residual (DLR), a training-only, parameter-free plug-in for low-rank pre-training that improves perplexity across LLaMA models from 60M to 7B parameters, and can be folded into the model after training with zero inference cost.

0 favorites 0 likes
#efficient-training

Frequency Domain Reservoir Computing

arXiv cs.LG ↗ · 2026-06-25 Cached

This paper introduces FRESCO, an Echo State Network architecture operating entirely in the frequency domain to achieve O(N) complexity for dense recurrent updates, matching state-of-the-art performance on benchmarks while reducing computational costs.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback