Tag
Introduces Nexus Sampling, a training-free KV-cache eviction method using weighted reservoir sampling instead of deterministic top-k, improving long-context LLM inference under fixed memory budgets, matching dense attention performance at 80% eviction.
VaSE is a training-free method for KV cache eviction that protects large-magnitude value states and introduces stochasticity to improve reasoning model accuracy under compression, outperforming existing methods.
Introduces triattention v3, a new attention mechanism that enables safe eviction without recall loss for long-context inference, demonstrated on a hybrid mamba+attention model up to 256k tokens.