Tag
A practical tutorial on measuring transformer inference performance, covering metrics like latency, TTFT, throughput, memory usage, and benchmarking techniques for LLMs.
An engineer's framework for understanding trade-offs in conversational AI systems between capability, control, and latency, illustrating why every assistant must choose two and suggesting deliberate design strategies.
OpenAI announces that audio now uses a dedicated fast path while reasoning and tool use run asynchronously, and voice-session startup has been reduced from six network round trips to one.
A guide on distinguishing whether latency in voice agents stems from the model or the network, helping developers identify bottlenecks.
Discusses 200 milliseconds as a critical threshold for response time, highlighting its role in perceived performance and user experience in tech systems.
An essay discussing the common statistical fallacy of averaging percentiles, framed with Hamlet quotes and Hacker News examples, and arguing for empathetic communication of statistical insights.
Liquid AI releases LFM2.5-Encoder-230M and LFM2.5-Encoder-350M, bidirectional encoders optimized for non-generative tasks like classification and retrieval, offering fast CPU inference at long context.
A blog post illustrating how relying solely on the mean can be misleading when evaluating performance improvements, using synthetic latency data to show the importance of looking at the full distribution via percentiles, density plots, and CDFs.
A practitioner recounts experiences with voice AI in call centers, detailing hidden costs when solutions underperform in production, and asks for honest feedback from others with similar real-world experience.
Baseten details how it built the fastest API for GLM-5.2, achieving over double the launch-day performance and introducing a latency-optimized Fast version for coding and agents, with further improvements planned.
Discusses the overlooked latency cost of decode speed in AI agent loops, affecting overall performance.
A discussion about whether customers dislike voice AI due to latency and pauses rather than the AI itself, based on experience with a banking contact center.
The article examines how a change in object state at an action chunk boundary can cause a one-chunk delay in decisions, and how tracking physical change, frame capture, model receipt, chunk commit, and first changed action can separate capture and transport delay from committed decisions, but still does not establish safety or generalization.
The article explains how prompt caching works in large language model agents, covering KV cache mechanics, prefill and decode phases, and the impact on latency, cost, and agent design.
The article explains why synthetic benchmarks for LLM inference can be misleading, as production traffic is bursty and variable, and recommends testing with real workloads to choose the right inference framework.
LangChain launches LangSmith tracing for voice frameworks (Pipecat, LiveKit, OpenAI Realtime, Gemini Live), enabling full audio monitoring, STT/TTS latency tracking, interruption detection, and VAD analysis with minimal code.
Octen is a web search tool that delivers full source-backed reports in under 3 minutes, outperforming OpenAI, Gemini, Grok, and Perplexity by 10-17 points on the DeepResearch Bench, combining speed and accuracy.
A curated list of text-to-speech resources for voice agent builders, organized around the decision between real-time streaming synthesis and offline high-fidelity synthesis, with emphasis on streaming latency and mid-stream cancellation.
A practical benchmark comparing Gemma dense (31B) and MoE (26B) models shows MoE is 25.5% faster and 20% cheaper per query with identical quality, validating theoretical cost savings.
Building an AI assistant that lives in iMessage without an interface reveals challenges with latency, acknowledgment, discoverability, and response length. A two-model pattern with a fast cheap pass for acknowledgment and a slow expensive pass for actual work solves the trust issue.