training-failures

#training-failures

What I learned building a debugger for PyTorch training loops and how it changed how I think about failure diagnosis [D]

Reddit r/MachineLearning ↗ · 2026-05-30

The author shares lessons from building NeuralDBG, an open-source debugger for PyTorch training loops that detects localized failures like vanishing/exploding gradients by monitoring per-layer gradient norm transitions instead of global loss. Practical code snippets and community questions are included.

0 favorites 0 likes

#training-failures

Notes on pretraining parallelisms and failed training runs (12 minute read)

TLDR AI ↗ · 2026-05-18 Cached

A technical deep-dive into common causes of failed pretraining runs in large language models, including causality-breaking issues in expert routing and numerical precision bugs, with examples from Llama 4, Gemini 2 Pro, and GPT-4.

0 favorites 0 likes

training-failures

What I learned building a debugger for PyTorch training loops and how it changed how I think about failure diagnosis [D]

Notes on pretraining parallelisms and failed training runs (12 minute read)

Submit Feedback