latency-optimization

提出 Speculative Sampling 策略，通过追加噪声潜变量扩展双向窗口并缓存预测结果，在不牺牲质量的前提下实现低延迟的交互式视频生成，使窗口大小成为质量与速度联合优化的决策变量。

TomasuLLM 是一个将乱序推测执行思想引入 LLM Agent 的运行时，让工具调用在隔离的写时复制沙箱中乱序执行并按序提交验证，从而在 SWE-bench Verified、Terminal-Bench 2.0 和 SWE-Marathon 上分别提升 1.31x、1.35x 和 1.27x 的性能，且在 4,010 条提交验证记录中零误报。

Augment Code switched its coding-agent backend to Stefano Ermon's Mercury 2.5 diffusion model, achieving 82% latency reduction and 90% cost cut in production. The article highlights the performance advantages of diffusion models and the need for independent AI benchmarking tools.

Introduces State-Grounded Conditioning (SGC), a design principle for user-facing LLM agents to condition on live state, reducing direction drift and improving latency and accuracy in in-game coaching scenarios.

An article critiques the overuse of AI agents for deterministic tasks, sharing a case study where a multi-agent customer support system was refactored with simpler code, resulting in lower latency and costs.

Agensh is a scalable self-organized multi-agent system without a central orchestrator that improves performance on complex tasks by scaling the number of agents, showing significant test-pass rate increases on benchmarks like ProgramBench and pandoc.

BrowserClaw is a lightweight Chrome automation platform that uses a dual-brain architecture to optimize browser interactions for AI agents, significantly reducing latency and token usage.

The article discusses key challenges in memory retrieval for voice agents, emphasizing the need for measuring P99 latency per turn, using prefetching, and budgeting memory tokens to reduce latency and improve user experience.

This paper introduces kernel-managed shared memory for AI systems, centralizing memory management to improve personalization and efficiency in multi-agent environments, with evaluations showing significant gains over alternative methods.

Arize Phoenix introduces M3, which adds sparse attention and a 1M-token context window to reduce latency and cost in AI agents by keeping long tool histories in context.

KeenableAI is announced as an AI-native search engine specifically for AI agents, featuring its own crawler, web index, and infrastructure to improve latency, quality, and cost efficiency.

The Pipecat team released PhoneLLM Alpha 1, an open-weights AI model optimized for low-latency voice agent applications, performing on par with larger models at a fraction of the cost. They also introduced PhoneBench v1, a benchmark for evaluating LLMs in phone agent scenarios.

Voice agents often face latency issues not mainly from the model but from components like endpointing; optimizing VAD and using proper benchmarking can reduce turn latency effectively.

Agnost AI launches its first model, agnost-*******-0.1, trained on customer production traces to outperform frontier models with major improvements in task success, latency, and cost, offering custom fine-tuning services.

SAGE is a learned SLO-aware adaptive retrieval policy for production RAG systems that dynamically selects the number of retrieved passages per query, improving SLO compliance and reducing latency/cost with minimal quality loss.

New research from UC Santa Barbara and LinkedIn introduces a self-speculating agent that unifies the agent and speculator roles in one model, improving next tool-call prediction accuracy (Hit@1) while preserving task success via joint agent-speculator reinforcement learning.

UltraViT is a latency-optimized vision encoder for large vision-language models, designed for on-device deployment with a pyramidal architecture and a two-stage generative pre-training strategy, achieving state-of-the-art performance at 1.7x speed.

FlashTrie presents a GPU-accelerated constrained beam search for generative retrieval, using a succinct trie layout and cooperative CUDA kernels to reduce decoding latency and enable real-time serving at scale, achieving up to 24× speedup and a 0.71% revenue lift in a commercial search engine.

A seasoned AI engineer shares key skills for 2026, including systematic output reading, context engineering, tool description discipline, eval design, model routing, prompt versioning, confidence scoring, streaming architecture, fallback chains, latency budgets, failure cataloguing, agent-vs-workflow decisions, and failure post-mortems as portfolio content.

Researchers trained a vision-language model without a vision encoder for only $100, inspired by Gemma 4 12B, achieving a 30% reduction in end-to-end latency on an M3 Pro MacBook.