Tag
Relativity Networks has raised $22 million to develop hollow-core fiber technology, which reduces latency by 30% and could transform data center geography for AI compute.
A benchmark study comparing MCP and filesystem access for AI agents in 20 production scenarios found that filesystem access reduces LLM costs by 27% and latency by 32% while improving answer quality.
This paper introduces elbow-based routing, a training-free inference-time method for MoE models that dynamically adjusts the number of active experts per token by detecting the elbow point in router probability distributions, achieving a 5.3% average latency reduction while maintaining accuracy.
Alibaba Cloud launches a one-stop AI memory solution combining PolarDB, MemOS, and MemTensor, offering relational, vector, and graph retrieval in a single PolarDB-PG instance, reducing P99 latency by up to 89.2%.
This paper presents a workload-aware cache eviction policy for multi-agent systems that uses recomputation cost, DAG dependency count, and agent invocation frequency to retain valuable cached entries, reducing latency by up to 64.7% over uncached baselines and 31.1% over the next best finite-capacity method.
FlashRT is an agent harness that guides coding agents to automatically optimize and deploy real-time multimodal applications, achieving up to 70x latency reduction on NVIDIA B200 GPUs and 3.6x throughput improvement on AMD MI355X.
This paper introduces Director, a distributed MoE serving system that minimizes end-to-end latency using prediction-driven, online proactive expert placement. It employs a lightweight predictor and a relaxation-based optimizer to achieve up to 55% latency reduction for models like Mistral, DeepSeek, and Qwen.
This article emphasizes the importance of LLM caching in production systems to reduce latency, GPU utilization, and costs, and introduces LMCache, an open-source KV cache management layer for scalable LLM inference.
Meta introduces techniques like Lazy Pre-Norm, Multi-CTA Norm Fusion, and FlashNormAttention to fuse normalization operations with GEMM and Attention kernels, hiding up to 90% of normalization latency on NVIDIA B200 hardware and achieving up to 35% latency reduction in attention blocks.
ProactiveLLM introduces a method for streaming LLMs to actively decide when to generate output based on endogenous cues, using mask-based streaming modeling and synchronized privileged self-distillation, reducing latency without external annotations.
Accio is a speculative execution framework that reduces cost and latency for web agents by leveraging offline site-structure profiling and online selection of fast paths, achieving a 1.9x reduction in per-task cost and 33.4% latency reduction while maintaining accuracy.
LiteFrame proposes a lightweight video encoder with Compressed Token Distillation training that reduces latency and enables processing 8x more frames for long-form video understanding in Video LLMs, improving accuracy while reducing compute.
LatentRAG is a novel framework that shifts reasoning and retrieval for agentic RAG into continuous latent space, reducing inference latency by approximately 90% while maintaining performance comparable to explicit methods.
OpenAI details how WebSockets and API optimizations reduced latency by 40% for agentic workflows, enabling GPT-5.3-Codex-Spark to reach near 1,000 tokens per second.
OpenAI introduces Prompt Caching, an automatic feature that reduces API costs by 50% and improves latency by reusing recently cached input tokens on GPT-4o, GPT-4o mini, o1-preview, and o1-mini models. The feature automatically applies to prompts longer than 1,024 tokens without requiring developer integration changes.