Tag
The author shares insights from training a small model with DeepSeek's latent attention, observing layer-dependent latent usage and a test-time trick that reduces KV cache 4x without loss change.
OjaKV introduces a context-aware online low-rank KV cache compression framework that uses hybrid storage and Oja's algorithm for incremental subspace adaptation to reduce GPU memory bottlenecks in long-context LLM inference without model fine-tuning.