Tag
Halcyon demonstrates that strong text-to-image generation can be trained from scratch on image-text pairs without large-scale language pretraining, outperforming existing models with far less compute. Controlled experiments show text-only pretraining does not improve T2I generation.
HRM-Text introduces a Hierarchical Recurrent Model that decouples computation into slow and fast layers, enabling efficient pretraining from scratch on only 40 billion tokens and a $1,500 budget, achieving competitive performance with larger models.
HRM-Text is a 1B parameter text generation model that uses a brain-inspired hierarchical recurrent architecture to achieve efficient pretraining with only 40B tokens and ~$1000, enabling accessible foundation model training with dramatically reduced compute and data requirements.