Tag
Proposes mechanism-driven monitors for preemptive detection of LLM training instability by deriving internal signals from low-precision flash attention and MoE routers, enabling detection thousands of steps before loss divergence.
A technique to accelerate matrix multiplication when M and N are small but K is large, as encountered in MoE routers and small-batch decoding, by decomposing K and running partial GEMMs in parallel. The approach beats PyTorch Inductor on most shapes using a custom Triton kernel.