latency

Tag

Cards List
#latency

@FinanceYF5: Kimi K3 went viral in two days. Deedy says there's a whole set of compute economics behind it. 1/ K3 has only been online for two days, but it has already reached #10 on OpenRouter, processing 140 billion tokens daily. However, the servers can't handle the load: throughput dropped from 30 Token/s to 13, latency spiked to 72 seconds, and time to first token exceeded 20 seconds...

X AI KOLs Following ↗ · 2026-07-20 Cached

Kimi K3 reached #10 on OpenRouter within two days of launch, processing 140 billion tokens daily, but high load caused throughput to drop and latency to spike.

0 favorites 0 likes
#latency

AI search API duel — SERPdive vs Tavily: ~60% of decided duels won, 20% fewer tokens, ~same latency

Reddit r/AI_Agents ↗ · 2026-07-20

Comparison of AI search APIs SERPdive and Tavily, showing SERPdive wins ~60% of comparisons with 20% fewer tokens and similar latency.

0 favorites 0 likes
#latency

How hard is it to build orbital data centers, actually?

Ars Technica ↗ · 2026-07-15 Cached

This Ars Technica article explores the technical challenges of building orbital data centers, focusing on SpaceX's plans for a constellation of AI1 satellites to power GPUs in space, covering heat dissipation, radiation, and latency issues.

0 favorites 0 likes
#latency

GPUHedge: Hedging serverless GPU providers improves cold start p95 latency from 117s to 30s [P]

Reddit r/MachineLearning ↗ · 2026-07-13

GPUHedge is an open-source tool that uses speculative execution to hedge between serverless GPU providers, reducing cold start p95 latency from 117s to 30s.

0 favorites 0 likes
#latency

@MaximeRivest: Much better latency and ability to edit by sketching! I have no idea if this: programming with a pen ...thing is going …

X AI KOLs Following ↗ · 2026-07-11 Cached

A tweet from MaximeRivest highlights improved latency and sketching capabilities for a pen-based programming tool, expressing excitement for its potential in data analytics and spreadsheets.

0 favorites 0 likes
#latency

Cut my agents' response latency 1.7× by switching to a model that thinks less — not one that decodes faster

Reddit r/AI_Agents ↗ · 2026-07-11

An article describing how to reduce AI agent response latency by 1.7× by switching to a model that requires less reasoning time rather than focusing on decoding speed.

0 favorites 0 likes
#latency

@googledevs: Upgrading your image pipeline? Meet Nano Banana 2 Lite! Engineered for rapid ideation and high-velocity developer pipel…

X AI KOLs Following ↗ · 2026-07-10 Cached

Google announces Nano Banana 2 Lite, an upgraded image pipeline model for rapid ideation and high-velocity developer pipelines, offering latency and performance improvements over the previous version.

0 favorites 0 likes
#latency

HydraDB OSS

Product Hunt ↗ · 2026-07-10 Cached

HydraDB OSS is an open-source graph database designed for AI applications, offering fast, low-latency performance with minimal operational overhead by leveraging object storage.

0 favorites 0 likes
#latency

Building a real-time AI tutor for 5-year-olds

Hacker News Top ↗ · 2026-07-09 Cached

Ello shares their engineering approach to building a real-time AI tutor for children ages 4-9, focusing on sub-second latency and pedagogy to avoid losing a child's attention.

0 favorites 0 likes
#latency

Memory in the Loop: In-Process Retrieval as ExtendedWorking Memory for Language Agents

arXiv cs.AI ↗ · 2026-07-08 Cached

This paper introduces 'memory in the loop', where language agents repeatedly access an in-process associative store on every reasoning step. By using a fast (∼100μs) in-process store, the per-step retrieval cost is reduced by three orders of magnitude compared to networked stores, eliminating redundant actions and improving recall across GPT-5-class models.

0 favorites 0 likes
#latency

Beyond Typing: The Architecture of Voice Vibing and Gesture Vibing

Reddit r/artificial ↗ · 2026-07-07 Cached

Analyzes the technical and economic barriers to voice coding, comparing token costs and latency, and predicts that gesture-based coding via XR headsets will become viable as hand-tracking latency drops below 30ms.

0 favorites 0 likes
#latency

Everyone Is Wrong About Open Source AI in the Enterprise (3 minute read)

TLDR AI ↗ · 2026-07-07 Cached

Decagon runs 90% of workloads on fine-tuned open-source models for latency and performance, while overall enterprise spending on open-source LLMs has dropped to 11% due to a surge in new use cases using frontier models. The article argues that as use cases mature, they will migrate from closed to open-source models.

0 favorites 0 likes
#latency

@msimoni: Re gRPC deadlines: what if the client says "I'm willing to wait 100ms" but the request takes 50ms on the network to rea…

X AI KOLs Timeline ↗ · 2026-07-06

A tweet raises a question about gRPC deadline propagation: if a client specifies a 100ms deadline but network latency consumes 50ms, does the server still have the full 100ms?

0 favorites 0 likes
#latency

@AstroHanRay: After comparison, Ollama Cloud's tokens/s is 40% higher than Z.ai's Max plan, and the first-token latency is much lower, with streaming output almost instant. Having tried various plans, I still think Ollama Cloud is the best for using GLM-5.2...

X AI KOLs Timeline ↗ · 2026-07-05 Cached

AstroHanRay compared Ollama Cloud and Z.ai Max plan and found Ollama Cloud significantly outperforms Z.ai in throughput and first-token latency, recommending Ollama Cloud for running the GLM-5.2 model.

0 favorites 0 likes
#latency

Wan-Streamer v0.2: Higher Resolution, Same Latency

Hugging Face Daily Papers ↗ · 2026-07-05 Cached

Wan-Streamer v0.2 is a latency-preserving upgrade to an end-to-end audio-visual interaction model, increasing output resolution from 192x336 to 640x368 while maintaining ~200 ms model-side latency via a multi-GPU thinker-performer architecture.

0 favorites 0 likes
#latency

Reducing LLM Latency

Reddit r/AI_Agents ↗ · 2026-07-03

Techniques and methods for reducing latency in large language models, improving inference speed.

0 favorites 0 likes
#latency

The Rollout Infrastructure Tax in Coding-Agent Reinforcement Learning

arXiv cs.LG ↗ · 2026-07-03 Cached

This paper studies the infrastructure overhead in coding-agent reinforcement learning, measuring up to 110x variation in cold-start latency and significant impact on training throughput across four execution substrates.

0 favorites 0 likes
#latency

@gabriel1: inference will be the biggest market in the world, intelligence is in infinite demand etched is bringing the AI Summer

X AI KOLs Timeline ↗ · 2026-06-30 Cached

Etched, an AI inference hardware startup, exited stealth after raising $800M and securing over $1B in customer contracts. Their first racks ship this summer, claiming state-of-the-art throughput, latency, and power efficiency.

0 favorites 0 likes
#latency

8 months running a voice agent in production: what broke, what fixed it, and the system prompt I use

Reddit r/AI_Agents ↗ · 2026-06-29

A practitioner shares 8 months of experience running a voice agent for a law firm, detailing challenges like latency, turn-taking, and post-call workflows, and provides a working system prompt.

0 favorites 0 likes
#latency

agent eval latency added 18 minutes to our CI. how are you running this without killing dev velocity?

Reddit r/AI_Agents ↗ · 2026-06-28

A discussion on the challenge of integrating comprehensive agent evaluations into CI, where latency from judge calls increases build time from 6 to 24 minutes, and potential solutions like parallelization, caching, and async eval are considered.

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback