latency

Anthropic 发布 Claude Platform 的 Message Threads 公测功能，通过仅发送增量消息而非重发完整对话历史，将长会话的时间到首 token（TTFT）中位数降低约 35%。

Anthropic is open-betaling "message threads" for the Claude Messages API: the server stores conversation history, so each turn developers only send new messages plus the previous reply ID. This significantly cuts latency on long conversations (median TTFT for 500K–1M token requests drops by ~35%), with pricing and rate limits unchanged.

The post argues that raw inference speed is becoming as important as model intelligence for AI agents, since agents run dozens of chained calls per task — comparing top throughput setups like N0.5 Flash (~2,000 tok/s), DeepSeek V4.1 Flash, GLM-5.3-Flash, and MiMo-V2.5-Pro UltraSpeed.

文章清晰解释了前缀感知的 LLM 路由机制：通过按块哈希提示词并维护块到 worker 的索引，让请求路由到持有共享前缀 KV 缓存的 GPU，实测在 8 个 vLLM Pod（16 张 H100）上将 p90 首 token 延迟从 92.5 秒降至 0.54 秒。

This paper proposes a small candidate formalizer trained on offline teacher-generated lists to bridge the interface gap between open-ended LLM workloads and fast decision models, reporting up to 52.6% task success on CALVIN-D with 2.09s end-to-end latency, compared to 11.8% without the formalizer and slower baselines like Qwen3.5-27B.

Garry Tan mentions efforts to slim down GStack to reduce token usage and task latency, with a reply noting CI pipeline times dropping from 7 to 3 minutes while asking about latency spikes from 20 running subagents.

Nace releases Drex Max 2.0, a 27B multimodal decision model for computer-use agents that scores options in a single forward pass with millisecond latency, priced at $0.06 per million input tokens via the Nace API.

Mouse-1 是一个面向 agent harness 的轻量决策模型，能处理 agent 执行过程中的大量小决策，宣称比 GPT-6 Sol 便宜 163 倍、快 65 倍，可与现有技术栈并行接入。

Qwen's 10-million-document RAG pipeline uses four lines of defense — hybrid retrieval, citation-constrained generation, claim-level verification, and calibrated abstention — to bring the hallucination rate on unanswerable questions down to 2%, while keeping p95 latency at 18ms. It's a production-grade anti-hallucination architecture worth studying.

A dev shares results from building a voice booking agent on ElevenLabs with AWS Lambda tools, achieving accurate pricing/availability but struggling with 5s P95 latency on tool turns and keeping booking steps ordered. The post asks the community how to reduce tool-turn latency, enforce step ordering, and make the post-call LLM judge reliable.

作者以语音 AI 编排平台产品负责人的视角，分析 Jev 及同类分类模型在电话语音机器人中的实际用途，并梳理了 Laya、Clef、CMF、Nimble、Kev 等竞品生态。文章指出单一 LLM 无法独立支撑生产级对话，分类模型在延迟、成本和结构化输出上仍不可或缺。

Coinbase announced a new matching engine for Coinbase Advanced that delivers 60x faster median order response and 480x faster tail latency, supporting trading of crypto, stocks, commodities, and FX.

Ali's open-source OpenSandbox added a fast sandbox feature that minimizes sandbox creation latency by bypassing Kubernetes scheduling and runtime. The author finds the architecture too hacky and prefers alternatives like E2B.

实践者分享构建电话语音AI代理的核心瓶颈并非模型能力，而是轮次切换时机：单靠VAD静音检测会导致打断或迟钝，改用STT置信度与部分转录文本的语义线索来判断发言是否结束效果更好。

该论文研究长上下文 LLM 服务中远程 KV 缓存层级的性能瓶颈，提出按并行 rank 分片 MLA KV 存储、异步加载等优化，将 p99 延迟从 83.32 秒降至 0.67 秒，并显著提升预填充吞吐。

A technical blog post comparing causal-decoder (v19) and encoder-based designs for strands-decider-2B, finding that encoders like e1b offer lower latency with similar accuracy, while encoder-decoder models scale worse in prompt length.

Sazabi 发布 1.0 版本，开放其自研日志摄取、存储与查询层，支持从终端、Web 应用或 AI 代理直接用自然语言查询日志，并宣称拥有全球最快的日志搜索速度。

A practitioner roundup of which LLMs teams are running for real-time voice agents, covering dedicated vs. serverless trade-offs (Gemma 4 31B, Qwen3 235B, GLM 5.2, Nemotron 3 Ultra, GPT-4.1/5.6, Gemini Flash), along with deployment lessons on measuring latency, caching, tool testing and idle GPU cost.

TacitVLA 提出一种无需在行动前生成推理文本的视觉-语言-动作模型，动作专家不关注其生成的推理内容，仅依靠推理监督改进骨干表示，在四个基准上以最高 3.4 倍更低延迟匹配显式推理模型的性能，并在 RoboMME 和 RMBench 上超越此前最佳结果。

Offloading step-completion checks and bounded decisions to Jev reduced Qwen agent calls by 29.6%, estimated API costs by 26.6%, and latency by 19.7%.