latency-optimization

The article discusses key challenges in memory retrieval for voice agents, emphasizing the need for measuring P99 latency per turn, using prefetching, and budgeting memory tokens to reduce latency and improve user experience.

This paper introduces kernel-managed shared memory for AI systems, centralizing memory management to improve personalization and efficiency in multi-agent environments, with evaluations showing significant gains over alternative methods.

Arize Phoenix introduces M3, which adds sparse attention and a 1M-token context window to reduce latency and cost in AI agents by keeping long tool histories in context.

KeenableAI is announced as an AI-native search engine specifically for AI agents, featuring its own crawler, web index, and infrastructure to improve latency, quality, and cost efficiency.

The Pipecat team released PhoneLLM Alpha 1, an open-weights AI model optimized for low-latency voice agent applications, performing on par with larger models at a fraction of the cost. They also introduced PhoneBench v1, a benchmark for evaluating LLMs in phone agent scenarios.

Voice agents often face latency issues not mainly from the model but from components like endpointing; optimizing VAD and using proper benchmarking can reduce turn latency effectively.

Agnost AI launches its first model, agnost-*******-0.1, trained on customer production traces to outperform frontier models with major improvements in task success, latency, and cost, offering custom fine-tuning services.

SAGE is a learned SLO-aware adaptive retrieval policy for production RAG systems that dynamically selects the number of retrieved passages per query, improving SLO compliance and reducing latency/cost with minimal quality loss.

New research from UC Santa Barbara and LinkedIn introduces a self-speculating agent that unifies the agent and speculator roles in one model, improving next tool-call prediction accuracy (Hit@1) while preserving task success via joint agent-speculator reinforcement learning.

UltraViT is a latency-optimized vision encoder for large vision-language models, designed for on-device deployment with a pyramidal architecture and a two-stage generative pre-training strategy, achieving state-of-the-art performance at 1.7x speed.

FlashTrie presents a GPU-accelerated constrained beam search for generative retrieval, using a succinct trie layout and cooperative CUDA kernels to reduce decoding latency and enable real-time serving at scale, achieving up to 24× speedup and a 0.71% revenue lift in a commercial search engine.

A seasoned AI engineer shares key skills for 2026, including systematic output reading, context engineering, tool description discipline, eval design, model routing, prompt versioning, confidence scoring, streaming architecture, fallback chains, latency budgets, failure cataloguing, agent-vs-workflow decisions, and failure post-mortems as portfolio content.

Researchers trained a vision-language model without a vision encoder for only $100, inspired by Gemma 4 12B, achieving a 30% reduction in end-to-end latency on an M3 Pro MacBook.

InfraMind introduces an infrastructure-aware multi-agent LLM orchestration framework that uses reinforcement learning to dynamically select models and topologies based on real-time system load, achieving up to 7x lower latency and 99.9% SLO compliance under high load.

ObjectCache proposes using S3-compatible object storage for LLM KV cache reuse to reduce cost and increase capacity, with a co-designed storage protocol and transfer schedule that minimizes latency overhead. Experiments show it adds only 5.6% latency over local DRAM for 64K contexts.

This paper introduces temporal semantic caching and MCP workflow optimizations for agentic plan-execute pipelines, achieving up to 30.6x speedup on cache hits and 1.67x overall speedup on the AssetOpsBench industrial benchmark.

A blog post detailing how to detect silent NPU fallback on Snapdragon in CI, including methods like running on real hardware, gating on coefficient of variation, and parsing ORT profiling JSON to identify fallen-back ops.

This paper introduces BoundaryRouter, a training-free framework that optimizes LLM agent usage by routing queries to either lightweight inference or full agent execution based on early experience. It also presents RouteBench, a benchmark for evaluating routing performance, showing significant improvements in speed and accuracy.