Tag
Explores techniques for measuring the correctness of semantic caches in production environments, a key concern for AI/ML systems relying on caching for efficiency.
Proposes SOLAR, a learning-augmented framework for semantic cache replacement in LLM agents, outperforming classical heuristics by using regret-based modification timing and Bayesian online learning. Achieves constant competitive ratio and significant improvements over FIFO.
Proposes building an open-source, lightweight semantic cache for LLMs using Rust/WASM at the CDN edge to reduce latency and API costs, seeking community feedback on architecture and use-case validity.
This paper proposes Semantic Cache Distillation (SCD), a loss-constrained framework that replaces raw KV cache transmission with compact semantic codes, achieving up to 2.65x TTFT speedup while keeping generation quality within 5% F1 of the oracle.