Tag
Shield Battery now adds fighting game-style rollback netcode to StarCraft: Brood War, turning the latency and stutters of the lockstep network model into nearly imperceptible frame skips and giving online matches near-local responsiveness.
shad0w is a tiny caching proxy that sits in front of any OpenAI-compatible local LLM server, learns the model's classification answers, and after ~1,000 samples compiles a table serving repeated queries in microseconds with 95%+ agreement guarantees. Apache-2.0, written in Python/JS/C, with a demo of it learning Snake from an LLM's moves.
该论文提出 PeopleMakingConversation 基准,通过 176 段非脚本对话中的 427 个停顿点和 482 个话轮结束点,评估六个实时语音系统在何时开口、延迟、回复长度与对话记忆等方面的自然对话表现,并公开数据与评测框架。
Garry Tan分享使用Capydotai驱动延迟与检索评估自动改进循环的体验,AI代理在过去16小时内自动运行并持续优化GBrain检索指标,无需人工干预。
微软宣布推出新型决策模型 Microsoft-Decision-1,专为快速决策任务设计,在结构化决策任务上于延迟和质量方面超越 LLM 及其他决策模型,目前已在事件响应、质量控制和科学发现等内部场景中测试。
OpenAI released a Decisions API, a fast "system 1"-style multimodal model built on GPT-6 Luna that classifies cropped images (e.g., eggs as clean, dirty, or cracked) with ~270ms latency, at roughly $0.03 per 1K image crops, comparable to RF-DETR on Roboflow.
This paper adapts a full-utterance speech language model for simultaneous speech translation using self-supervised prefix translations, showing that multi-turn append-only decoding with confidence-based commit control improves quality-latency frontiers and sharply reduces commit-calibration error on FLEURS and CoVoST 2.
JevResearch presents an independent, live evaluation of TypeSafe AI's Jev model across ~16k benchmark requests, concluding it is a cheap, low-latency sub-frontier reasoner best suited for routing and rubric-grading tasks rather than frontier-class reasoning.
StreamDecisionBench (SDB) evaluates language models as real-time decision components inside applications, measuring the 'decision in force' over evolving evidence streams and attributing errors to judgment, latency, or both. Across four hosted models, it finds that reasoning improves judgment but costs substantial latency, with one faster, weaker component achieving an integrated score comparable to a stronger non-reasoning model.
Eric Zhang shares a steady-state analysis arguing that PD disaggregation mainly improves mean per-token latency for prefill-heavy workloads (giving ~1/(decode-time fraction) interactivity gains) and is largely unnecessary for decode-bound, low-latency inference — validated in production, and echoed by Horace He's remark that few have really thought through PD disagg vs. mixed-chunk prefill.
A tweet criticizing the slow performance of cloud AI models compared to self-hosted solutions.
A tweet discussing a claim of 80ms latency in Voice AI systems, raising questions about its feasibility with custom models and on-prem inference.
Contrastive Language Models (CLM) is introduced as an open-weights alternative to TypeSafe AI's JEV, offering functional parity with improved latency and fine-tuning capabilities, though with trade-offs in generalization.
The author explains how they optimized a delivery service's routing engine latency by leveraging Redis hash slots to handle batch operations correctly in a Redis cluster.
A developer shares their achievement of reducing object detection latency to below 0.35 seconds on local hardware, highlighting progress in AI performance optimization.
The author discusses design approaches for routing AI agent requests when model capability, policy, cost, and latency conflict, asking for trade-offs and strategies from production experience.
The tweet highlights the low latency and scalability of calling Jev from the US West Coast, with 130 ms per request for 6 questions and constant latency under high parallelism, indicating good design and future potential with specialized models.
The author compared TypeSafe AI's Jev model with a regular LLM for model routing and found that Jev significantly reduces latency to around 1 second versus 4-14 seconds, making it promising for fast decision layers.
Arize Phoenix promotes a tool offering low latency (70-500 ms), zero type errors, calibrated probabilities, and low cost ($0.042/MTok), emphasizing the need for observability in code decisions.
The tweet identifies inference serving as a prime target for autoresearch, emphasizing end-to-end optimization with constraints on latency, quality, and throughput, covering various aspects in a unified search space and hinting at future developments.