My learnings from optimizing training pipeline to go from 36 steps/minute to 47 steps/minute
Summary
The author shares techniques that improved training pipeline performance from 36 to 47 steps per minute.
Similar Articles
The $110/month self-improving pipeline (5 minute read)
A developer shares their $110/month automated pipeline that uses Claude AI to triage, decompose, implement, and test GitHub issues, resulting in 27 merges over 2 weeks with minimal failures.
A Single Rewrite Suffices: Empirical Lessons from Production Skill Description Optimization
This paper presents an automated pipeline for optimizing natural language skill descriptions in enterprise AI agents to resolve skill collisions, achieving performance matching manual tuning with a 32× speedup. Ablation studies show that a single LLM rewrite using error cases captures most improvements, while other design choices have minimal impact.
Turning local agents into self-optimizing agents
A self-optimizing agentic pipeline that improves benchmark performance from ~30% to ~90% on TerminalBench, and can be extended to everyday chats by logging interactions, reflecting with a local model, and injecting lessons into future system prompts.
One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining
This paper challenges the assumption that one-step gradient delay in asynchronous pipeline parallelism is inherently unstable, showing that degradation depends on optimizer choice. It demonstrates that optimizers like Muon are robust to one-step delay and introduces an error-feedback correction to further mitigate staleness, achieving near-synchronous performance in LLM pretraining up to 10B parameters.
@yukangchen_: Excited to share our new blog: Scaling Video Training with Parallelism https://research.nvidia.com/labs/eai/blogs/scali…
This blog from NVIDIA Research discusses how sequence parallelism can scale long-video training systems for both understanding and generation, addressing the challenge of fitting very long video sequences across multiple GPUs.