Tag
Kimi K3 reached #10 on OpenRouter within two days of launch, processing 140 billion tokens daily, but high load caused throughput to drop and latency to spike.
Comparison of AI search APIs SERPdive and Tavily, showing SERPdive wins ~60% of comparisons with 20% fewer tokens and similar latency.
This Ars Technica article explores the technical challenges of building orbital data centers, focusing on SpaceX's plans for a constellation of AI1 satellites to power GPUs in space, covering heat dissipation, radiation, and latency issues.
GPUHedge is an open-source tool that uses speculative execution to hedge between serverless GPU providers, reducing cold start p95 latency from 117s to 30s.
A tweet from MaximeRivest highlights improved latency and sketching capabilities for a pen-based programming tool, expressing excitement for its potential in data analytics and spreadsheets.
An article describing how to reduce AI agent response latency by 1.7× by switching to a model that requires less reasoning time rather than focusing on decoding speed.
Google announces Nano Banana 2 Lite, an upgraded image pipeline model for rapid ideation and high-velocity developer pipelines, offering latency and performance improvements over the previous version.
HydraDB OSS is an open-source graph database designed for AI applications, offering fast, low-latency performance with minimal operational overhead by leveraging object storage.
Ello shares their engineering approach to building a real-time AI tutor for children ages 4-9, focusing on sub-second latency and pedagogy to avoid losing a child's attention.
This paper introduces 'memory in the loop', where language agents repeatedly access an in-process associative store on every reasoning step. By using a fast (∼100μs) in-process store, the per-step retrieval cost is reduced by three orders of magnitude compared to networked stores, eliminating redundant actions and improving recall across GPT-5-class models.
Analyzes the technical and economic barriers to voice coding, comparing token costs and latency, and predicts that gesture-based coding via XR headsets will become viable as hand-tracking latency drops below 30ms.
Decagon runs 90% of workloads on fine-tuned open-source models for latency and performance, while overall enterprise spending on open-source LLMs has dropped to 11% due to a surge in new use cases using frontier models. The article argues that as use cases mature, they will migrate from closed to open-source models.
A tweet raises a question about gRPC deadline propagation: if a client specifies a 100ms deadline but network latency consumes 50ms, does the server still have the full 100ms?
AstroHanRay compared Ollama Cloud and Z.ai Max plan and found Ollama Cloud significantly outperforms Z.ai in throughput and first-token latency, recommending Ollama Cloud for running the GLM-5.2 model.
Wan-Streamer v0.2 is a latency-preserving upgrade to an end-to-end audio-visual interaction model, increasing output resolution from 192x336 to 640x368 while maintaining ~200 ms model-side latency via a multi-GPU thinker-performer architecture.
Techniques and methods for reducing latency in large language models, improving inference speed.
This paper studies the infrastructure overhead in coding-agent reinforcement learning, measuring up to 110x variation in cold-start latency and significant impact on training throughput across four execution substrates.
Etched, an AI inference hardware startup, exited stealth after raising $800M and securing over $1B in customer contracts. Their first racks ship this summer, claiming state-of-the-art throughput, latency, and power efficiency.
A practitioner shares 8 months of experience running a voice agent for a law firm, detailing challenges like latency, turn-taking, and post-call workflows, and provides a working system prompt.
A discussion on the challenge of integrating comprehensive agent evaluations into CI, where latency from judge calls increases build time from 6 to 24 minutes, and potential solutions like parallelization, caching, and async eval are considered.