Tag
HyperParallel is a PyTorch-based distributed acceleration library optimized for Ascend SuperPoD, demonstrating higher training throughput than standard PyTorch FSDP2 with Muon while maintaining loss convergence, as showcased in a demo at #PyTorchCon China.
KDFlow is a novel knowledge distillation framework for large language models that uses a decoupled architecture with SGLang for teacher inference and FSDP2 for student training, achieving 1.44x to 6.36x speedup over existing frameworks.
HRM-Text is a 1B parameter text generation model that uses a brain-inspired hierarchical recurrent architecture to achieve efficient pretraining with only 40B tokens and ~$1000, enabling accessible foundation model training with dramatically reduced compute and data requirements.