Tag
The paper studies how 60 language models from 13 vendors behave when users apply pressure, finding that folding rates correlate with model recency while holding manner varies by vendor. It also shows LLM coders can consistently apply human-authored codebooks, indicating humans should focus on defining behaviors rather than labeling volume.
StableVQ proposes practical guidelines to stabilize the training of vector-quantized tokenizers by decoupling encoder-decoder and codebook training, improving stability and codebook utilization for image generation models.
This paper proposes Hyper-Spherical Quantization (HSQ) to address codebook collapse in discretizing visual representations, achieving high-fidelity reconstruction and scalable codebook budgets up to 131,072 with 100% utilization.
Gefen is a memory-efficient optimizer that reduces AdamW's memory footprint by ~8x by automatically sharing second-moment estimates and quantizing the first moment using a learned codebook, while maintaining AdamW-level performance.