lm-head

ARCHead is a packed LM-head compressor that combines quantized low-rank factors, INT4 residuals, and an activation-metric correction to reduce storage by ~3.7-3.9x while preserving perplexity, complementing existing block quantizers.

The paper introduces Ember, a lightweight optimizer for embedding and LM-head matrices that exploits gradient geometry to improve efficiency and performance across supervised finetuning, RL, and pretraining, while using far less optimizer state than Adam.