Tag
This article emphasizes the importance of LLM caching in production systems to reduce latency, GPU utilization, and costs, and introduces LMCache, an open-source KV cache management layer for scalable LLM inference.
Researchers introduce Programmable KV Cache, a method for editing and composing KV caches to avoid re-prefilling long contexts during LLM agent inference, achieving 53–398× reduction in p90 time-to-first-token while maintaining decision identity.
A developer recounts the painful experience of building and eventually shutting down a production LLM-based service for medical appointment scheduling, highlighting issues with model reliability, structured output validation, and provider uptime.
This paper presents Decoupled Search Grounding (DSG), a vendor-agnostic architecture that separates search retrieval from LLM reasoning, enabling explicit control over provider routing, caching, and output contracts. Experiments show DSG nearly matches native search accuracy at 91% lower cost and 68% lower latency.
The author compiled a glossary of confusing LLM terms with production-oriented explanations, cleaned it up, and open-sourced it as a browsable UI on GitHub.
A production LLM systematically repurposes tool schema enums to invent helpful UI buttons across 2,400 messages, showing strategic deviation from constraints that improves UX rather than causing harm.