batching

This paper characterizes the serving behavior of masked diffusion language models (dLLMs) using real hardware measurements, identifying key differences from autoregressive models and deriving design principles for efficient inference systems.

The author experimented with two local agents running in parallel on a MacBook Pro M3Max using the QWEN 3.8 27B 4bit model, finding that batching enables concurrent execution but increases latency, with an optimal agent count around 4.

Opinion piece arguing that local AI models will never win because they are weaker, more expensive, and less efficient than datacenter inference, due to batching and GPU advantages.

pgrust 0.2 rebuilds the Postgres query engine with batching, operator fusion, and SIMD, achieving 300x faster analytical queries and 30% faster OLTP than Postgres.

BatchDAG introduces a system where an LLM generates typed directed acyclic graphs of operations for scalable ad-hoc analysis over enterprise data, achieving up to 47x reduction in LLM calls and sub-60-second query times over 50,000+ meetings.

This documentation introduces hijax types, a new feature in JAX that allows defining custom types with their own invariants, tangent types, batching, and sharding behavior, illustrated with an example of quantized arrays.

This article explains how batching and parallel operations improve latency and efficiency in AI computer use systems, highlighting open-source implementations like pi-computer-use and cua-driver that achieved significant performance gains before similar features appeared in Codex.

A detailed thread explaining key concepts of LLM inference: attention, KV caching, chunked prefill, and batching techniques, including continuous batching used in vLLM and SGLang.

An open, in-progress handbook explaining LLM inference internals including GPU memory hierarchy, KV cache, batching, and popular inference engines like vLLM and TensorRT-LLM.

Setting temperature to 0 does not guarantee deterministic tool calls in agents due to batched inference causing floating-point reduction order shifts, leading to token flips and different actions under load.

A study by LangChain and Harvey explores methods to reduce the cost of verifying legal agent outputs by batching criteria evaluations and using open models, achieving order-of-magnitude cost savings while maintaining near-frontier performance.

GreptimeDB v1.0 introduces Pending Rows Batcher, a three-stage pipeline that moves CPU-intensive work off the Datanode's critical section, improving Prometheus remote write throughput from 1.20M to 2.17M points/sec and reducing Datanode CPU usage by 20%.

This paper analyzes the trade-off between mixed batching and exclusive batching for LLM inference, showing that the optimal choice depends on GPU memory bandwidth. It proposes a threshold-based hybrid scheduler that dynamically switches between the two methods, achieving up to 41.9% higher throughput on bandwidth-constrained GPUs.

Demonstrates running subagents locally on a MacBook Pro M5 using Codex CLI and LM Studio with Qwen 3.6 and MLX batching for code review and bug detection.

LM Studio announces a beta update to its MLX engine, introducing batching for vision models and improved caching for faster inference.