pause-tokens

AbridgeKV trains models to compress completed blocks of generated chain-of-thought text into the KV entries of a few learned pause tokens, achieving a 3.5× smaller cache and 3.2× faster decoding on a saturated H100 with no accuracy loss at a quarter of the full KV budget.