Tag
The author experimented with KV cache blending by splitting prompts into chunks with overlap, achieving a 3x boost in prefill speed without affecting retrieval tasks on Ling3-tiny.
An extensive benchmark of 13 local LLMs at 65K-128K context shows that prefill speed dominates agentic workload performance (94-99% of wall-clock time), rendering tg128 misleading, and that KV head count is the key architectural factor over parameter count or MoE/dense design.
This article explains that for RAG applications, prefill speed matters more than decode speed, and discusses why AMD's Strix Halo APU struggles with interactive use cases.