batching

BatchDAG introduces a system where an LLM generates typed directed acyclic graphs of operations for scalable ad-hoc analysis over enterprise data, achieving up to 47x reduction in LLM calls and sub-60-second query times over 50,000+ meetings.

This documentation introduces hijax types, a new feature in JAX that allows defining custom types with their own invariants, tangent types, batching, and sharding behavior, illustrated with an example of quantized arrays.

This article explains how batching and parallel operations improve latency and efficiency in AI computer use systems, highlighting open-source implementations like pi-computer-use and cua-driver that achieved significant performance gains before similar features appeared in Codex.

A detailed thread explaining key concepts of LLM inference: attention, KV caching, chunked prefill, and batching techniques, including continuous batching used in vLLM and SGLang.

An open, in-progress handbook explaining LLM inference internals including GPU memory hierarchy, KV cache, batching, and popular inference engines like vLLM and TensorRT-LLM.

Setting temperature to 0 does not guarantee deterministic tool calls in agents due to batched inference causing floating-point reduction order shifts, leading to token flips and different actions under load.

A study by LangChain and Harvey explores methods to reduce the cost of verifying legal agent outputs by batching criteria evaluations and using open models, achieving order-of-magnitude cost savings while maintaining near-frontier performance.

GreptimeDB v1.0 introduces Pending Rows Batcher, a three-stage pipeline that moves CPU-intensive work off the Datanode's critical section, improving Prometheus remote write throughput from 1.20M to 2.17M points/sec and reducing Datanode CPU usage by 20%.

This paper analyzes the trade-off between mixed batching and exclusive batching for LLM inference, showing that the optimal choice depends on GPU memory bandwidth. It proposes a threshold-based hybrid scheduler that dynamically switches between the two methods, achieving up to 41.9% higher throughput on bandwidth-constrained GPUs.

Demonstrates running subagents locally on a MacBook Pro M5 using Codex CLI and LM Studio with Qwen 3.6 and MLX batching for code review and bug detection.

LM Studio announces a beta update to its MLX engine, introducing batching for vision models and improved caching for faster inference.