Tag
KV Packet proposes a recomputation-free cache reuse framework for LLMs that uses trainable soft-token adapters to bridge context discontinuities, eliminating overhead while maintaining performance comparable to full recomputation baselines on Llama-3.1 and Qwen2.5.
oMLX is a new open-source tool for optimized LLM inference on Apple Silicon Macs, featuring continuous batching and tiered KV caching managed via a menu bar app.