Tag
Anthropic announces pricing changes for its AI model API, with rates of $10 and $50 per million input and output tokens respectively, and a 75% reduction in cache read costs. The company claims these changes lead to about 25% lower typical costs and up to 45% savings for agentic workloads.
AMD is working hard to improve its software for AI workloads, especially agentic tasks, with potential for better performance per dollar than NVIDIA, but software and leadership issues currently hinder progress.
A blog post explaining that cache read costs dominate LLM inference spending for agentic workloads, with cumulative costs growing quadratically as context is re-read each turn, and advice to reduce tool call count to cut costs.
TokTier is a stateful tokenization service that cuts tokenization overhead in agentic LLM serving by reusing stable token boundaries and running exact CPU/GPU tokenization, reducing time-to-first-token by 16–34% under vLLM.
Describes a method for unifying outputs from thousands of agents in parallel forecasting tasks, achieving consistency and cost efficiency through post-processing and homogeneous task design.
Mooncake announces support for SSD offloading of KV cache, enabling cost-effective scaling of KV cache capacity beyond DRAM for long-lived agentic workloads, with analysis showing bimodal reuse patterns that make tiered storage efficient.
An extensive benchmark of 13 local LLMs at 65K-128K context shows that prefill speed dominates agentic workload performance (94-99% of wall-clock time), rendering tg128 misleading, and that KV head count is the key architectural factor over parameter count or MoE/dense design.
DualPath is a system that breaks the storage bandwidth bottleneck in agentic LLM inference by introducing a dual-path KV-cache loading mechanism, improving throughput by up to 1.87x offline and 1.96x online.
vLLM integrates Mooncake Store for distributed KV cache reuse, enabling cross-node prefix caching to efficiently serve agentic workloads with high token reuse.
Qwen inference team announced TokenSpeed, a high-performance LLM inference engine for agentic workloads, achieving 540 TPS, with open-source preview available.
Lightseek releases TokenSpeed, a high-performance LLM inference engine optimized for agentic workloads, featuring compiler-backed parallelism and advanced kernel optimizations that have been adopted by vLLM.