training-throughput

At an AI infrastructure meetup, multiple people reported that GPU clusters often sit idle because storage systems can't feed data fast enough during training, especially with large unstructured datasets on older NAS. Attendees mentioned moving to high-throughput S3-optimized platforms like Cloudian HyperStore and VAST Data to address the bottleneck.

This paper presents AsyncOPD, a fully asynchronous on-policy distillation pipeline for LLMs, systematically studying the effects of stale-policy data and proposing estimator designs that improve training throughput by 1.6-3.8x while maintaining comparable accuracy.