@shikhargupta02: I’ve been learning about latent attention (by deepseek). Instead of storing a full K and a V vector per token, it rathe…

I’ve been learning about latent attention (by deepseek). Instead of storing a full K and a V vector per token, it rather stores a common latent vector L for both - using a learned low rank projection of the input. K and V can be constructed from L by projecting it up (again, learned). The dimension of this latent vector << total K+V storage per token. So naturally, the stored KV cache goes down. Some interesting patterns emerged when I trained a small model with it. The first and the final few layers rely on the latent space a lot more than the intermediate layers. Layers 5-7 use only 25% of the 256 dimensions available. They seem to need less complexity in their K and V vectors. The first layer, on the other hand, uses the latent space the most which makes sense as it is closest to the embedding layer and has a lot of raw signal to encode. I did a test time experiment where I keep only the most prominent latent dims and shave off the rest. There was almost no change in the validation loss while reducing the kv cache 4x compared to full rank latent vector (which was already 6x less compared to multi head attention). A very cool test time optimization.