latency

Tag

Cards List
#latency

We Put Fighting Game Netcode in Brood War

Lobsters Hottest ↗ · 2026-10-12 Cached

Shield Battery now adds fighting game-style rollback netcode to StarCraft: Brood War, turning the latency and stutters of the lockstep network model into nearly imperceptible frame skips and giving online matches near-local responsiveness.

0 favorites 0 likes
#latency

I built a tiny table that learns your local LLM's classification answers and serves the repeats in microseconds [D]

Reddit r/MachineLearning ↗ · 2026-10-12

shad0w is a tiny caching proxy that sits in front of any OpenAI-compatible local LLM server, learns the model's classification answers, and after ~1,000 samples compiles a table serving repeated queries in microseconds with 95%+ agreement guarantees. Apache-2.0, written in Python/JS/C, with a demo of it learning Snake from an LLM's moves.

0 favorites 0 likes
#latency

People Making Conversation: Evaluating How Naturally Speech Models Converse

arXiv cs.CL ↗ · 2026-10-12 Cached

该论文提出 PeopleMakingConversation 基准,通过 176 段非脚本对话中的 427 个停顿点和 482 个话轮结束点,评估六个实时语音系统在何时开口、延迟、回复长度与对话记忆等方面的自然对话表现,并公开数据与评测框架。

0 favorites 0 likes
#latency

@garrytan: I'm amazed how simple it is to be driving new features and loops on latency/retrieval eval auto-improvement using @capy…

X AI KOLs Timeline ↗ · 2026-10-10 Cached

Garry Tan分享使用Capydotai驱动延迟与检索评估自动改进循环的体验,AI代理在过去16小时内自动运行并持续优化GBrain检索指标,无需人工干预。

0 favorites 0 likes
#latency

@satyanadella: Introducing Microsoft-Decision-1, our new model for fast decision-making. It delivers top performance on structured dec…

X AI KOLs Timeline ↗ · 2026-10-09 Cached

微软宣布推出新型决策模型 Microsoft-Decision-1,专为快速决策任务设计,在结构化决策任务上于延迟和质量方面超越 LLM 及其他决策模型,目前已在事件响应、质量控制和科学发现等内部场景中测试。

0 favorites 0 likes
#latency

@erik_kokalj: openai just released Decisions API, Jev-like "system 1" multi-modal model based on GPT6 Luna! Each egg gets cropped and…

X AI KOLs Timeline ↗ · 2026-10-07 Cached

OpenAI released a Decisions API, a fast "system 1"-style multimodal model built on GPT-6 Luna that classifies cropped images (e.g., eggs as clean, dirty, or cracked) with ~270ms latency, at roughly $0.03 per 1K image crops, comparable to RF-DETR on Roboflow.

0 favorites 0 likes
#latency

Learning When to Commit from Partial Speech for End-to-End Simultaneous Speech Translation

arXiv cs.CL ↗ · 2026-10-05 Cached

This paper adapts a full-utterance speech language model for simultaneous speech translation using self-supervised prefix translations, showing that multi-turn append-only decoding with confidence-based commit control improves quality-latency frontiers and sharply reduces commit-calibration error on FLEURS and CoVoST 2.

0 favorites 0 likes
#latency

Jev: Not Frontier, But Still Worth Your Attention [R]

Reddit r/MachineLearning ↗ · 2026-10-03 Cached

JevResearch presents an independent, live evaluation of TypeSafe AI's Jev model across ~16k benchmark requests, concluding it is a cheap, low-latency sub-frontier reasoner best suited for routing and rubric-grading tasks rather than frontier-class reasoning.

0 favorites 0 likes
#latency

StreamDecisionBench: Evaluating Decisions in Force on Evolving Language Streams

arXiv cs.CL ↗ · 2026-10-01 Cached

StreamDecisionBench (SDB) evaluates language models as real-time decision components inside applications, measuring the 'decision in force' over evolving evidence streams and attributing errors to judgment, latency, or both. Across four hosted models, it finds that reasoning improves judgment but costs substantial latency, with one faster, weaker component achieving an integrated score comparable to a stronger non-reasoning model.

0 favorites 0 likes
#latency

@ekzhang1: So I would like to share my analysis of PD disagg that I've run by Horace and many folks at Thinky You may have heard t…

X AI KOLs Timeline ↗ · 2026-09-30 Cached

Eric Zhang shares a steady-state analysis arguing that PD disaggregation mainly improves mean per-token latency for prefill-heavy workloads (giving ~1/(decode-time fraction) interactivity gains) and is largely unnecessary for decode-bound, low-latency inference — validated in production, and echoed by Horace He's remark that few have really thought through PD disagg vs. mixed-chunk prefill.

0 favorites 0 likes
#latency

@TheAhmadOsman: Seriously how could people tolerate these slow ass cloud models anymore Painfully slow in comparison to anything I self…

X AI KOLs Timeline ↗ · 2026-09-28 Cached

A tweet criticizing the slow performance of cloud AI models compared to self-hosted solutions.

0 favorites 0 likes
#latency

Best case Voice AI Latency

Reddit r/AI_Agents ↗ · 2026-09-25

A tweet discussing a claim of 80ms latency in Voice AI systems, raising questions about its feasibility with custom models and on-prem inference.

0 favorites 0 likes
#latency

JEV almost dead: CLM vs JEV

Reddit r/LocalLLaMA ↗ · 2026-09-24

Contrastive Language Models (CLM) is introduced as an open-weights alternative to TypeSafe AI's JEV, offering functional parity with improved latency and fine-tuning capabilities, though with trade-offs in generalization.

0 favorites 0 likes
#latency

Why didn't anybody tell me about Redis hash slots?

Lobsters Hottest ↗ · 2026-09-23 Cached

The author explains how they optimized a delivery service's routing engine latency by leveraging Redis hash slots to handle batch operations correctly in a Redis cluster.

0 favorites 0 likes
#latency

@yoheinakajima: got object detection down to below 0.35 sec latency locally

X AI KOLs Timeline ↗ · 2026-09-23 Cached

A developer shares their achievement of reducing object detection latency to below 0.35 seconds on local hardware, highlighting progress in AI performance optimization.

0 favorites 0 likes
#latency

How do you route agent requests when model capability, policy, cost, and latency conflict?

Reddit r/AI_Agents ↗ · 2026-09-23

The author discusses design approaches for routing AI agent requests when model capability, policy, cost, and latency conflict, asking for trade-offs and strategies from production experience.

0 favorites 0 likes
#latency

@taroleo: When calling Jev from the US West Coast, a single request compiling 6 questions takes about 130 ms, or 20-25 ms per jud…

X AI KOLs Timeline ↗ · 2026-09-21

The tweet highlights the low latency and scalability of calling Jev from the US West Coast, with 130 ms per request for 6 questions and constant latency under high parallelism, indicating good design and future potential with specialized models.

0 favorites 0 likes
#latency

Tried TypeSafe AI’s Jev vs a regular LLM for model routing and the latency difference is pretty noticeable

Reddit r/AI_Agents ↗ · 2026-09-20

The author compared TypeSafe AI's Jev model with a regular LLM for model routing and found that Jev significantly reduces latency to around 1 second versus 4-14 seconds, making it promising for fast decision layers.

0 favorites 0 likes
#latency

@ArizePhoenix: 70 to 500 ms, 0% type errors, calibrated probabilities on every answer, $0.042/MTok in and output free. When decisions …

X AI KOLs Following ↗ · 2026-09-18 Cached

Arize Phoenix promotes a tool offering low latency (70-500 ms), zero type errors, calibrated probabilities, and low cost ($0.042/MTok), emphasizing the need for observability in code decisions.

0 favorites 0 likes
#latency

@cyrusasg: inference serving is one of the cleanest targets for autoresearch. a lot of the attention right now is kernel gen, but …

X AI KOLs Timeline ↗ · 2026-09-15 Cached

The tweet identifies inference serving as a prime target for autoresearch, emphasizing end-to-end optimization with constraints on latency, quality, and throughput, covering various aspects in a unified search space and hinting at future developments.

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback