Tag
HotKV：一个用 C++20 从零编写的多线程、Redis 兼容的内存存储，内置面向 LLM 响应的语义缓存。在 Banking77 基准上对 Claude 系列模型测试显示，可将调用、Token 和成本减少约 70%，同时详细分析了语义缓存对答案质量的影响。
Redis LangCache is a semantic caching tool that reduces LLM costs by up to 70% by storing and reusing similar question-response pairs, making AI applications faster and more cost-effective.
Explores techniques for measuring the correctness of semantic caches in production environments, a key concern for AI/ML systems relying on caching for efficiency.
Proposes SOLAR, a learning-augmented framework for semantic cache replacement in LLM agents, outperforming classical heuristics by using regret-based modification timing and Bayesian online learning. Achieves constant competitive ratio and significant improvements over FIFO.
Proposes building an open-source, lightweight semantic cache for LLMs using Rust/WASM at the CDN edge to reduce latency and API costs, seeking community feedback on architecture and use-case validity.
This paper proposes Semantic Cache Distillation (SCD), a loss-constrained framework that replaces raw KV cache transmission with compact semantic codes, achieving up to 2.65x TTFT speedup while keeping generation quality within 5% F1 of the oracle.