inference-optimization

This post describes two configuration flags that increase the official Ling-3.0-flash INT4 inference speed from 20.8 to 38.7 tok/s on a single DGX Spark, while warning about the need for a specific vLLM fork and noting tradeoffs with long-context performance.

A developer benchmarks Qwen3.6-35B-A3B Q6 on an RTX 3090, showing that offloading eight MoE expert layers to CPU and increasing batch sizes improves prompt processing by 2.36× (564→1330 tok/s) with no decode speed regression, using evolutionary search to find the tuning config.

This paper introduces elbow-based routing, a training-free inference-time method for MoE models that dynamically adjusts the number of active experts per token by detecting the elbow point in router probability distributions, achieving a 5.3% average latency reduction while maintaining accuracy.

This empirical study examines practical online KV cache compaction for LLM agents, comparing token eviction and attention matching methods under different proxy query sources. It finds that delaying compaction to use future agent queries recovers performance, and token eviction preserves accuracy while reducing KV cache by 80%.

The VIDRAFT team shares their verified state-of-the-art inference optimization recipe for the Fast Gemma Challenge, achieving 510.58 TPS on a single A10G with PPL 2.39 using a fully public vLLM-based config.

Cloudflare details how it serves large open-source MoE models like Kimi K2.6 and GLM 5.2 efficiently using FP8 KV cache quantization and weight compression, boosting throughput and cutting costs without accuracy loss.

BLADE is a lightweight framework that dynamically terminates LLM reasoning by expanding probe checkpoints to sentence, self-doubt, and paragraph boundaries, while adaptively selecting informative hidden layers. Experiments on Qwen3 models show near-baseline accuracy with 24.8% token reduction on Qwen3-8B and 15.8% on Qwen3-4B.

Wafer announces that it can serve Kimi K3 on AMD MI355X at 3.8x higher throughput and 71% lower cost than on B200 nodes, arguing that AMD's large VRAM and software support make it the best performance-per-dollar choice for frontier models.

This post shares fixes to improve DeepSeek v4 Flash prefill/PP speed: downgrading CUDA from 13.3 to 13.1 or using a custom fork, achieving up to 1.3K prompt processing tokens/s.

Kimi K3 by Moonshot is an open-weight model ranking fourth among 580 models, featuring innovations like Kimi Delta Attention to reduce KV cache memory, Quantile Balancing for expert load balancing, and AgentENV for efficient RL training sandboxing.

OpenAI's blog post describes how GPT-5.6 Sol, a new frontier model, uses self-optimization to improve its own inference efficiency while maintaining high intelligence.

OpenAI announces the GPT-5.6 model family, including Sol, Terra, and Luna, which achieve frontier intelligence with significantly improved efficiency and cost reductions, backed by innovations in inference and agentic harness.

Eigen Labs launches an open autoresearch competition called MLX.fast to optimize inference speed of the Laguna XS 2.1 model on consumer Macs, aiming to make it as fast as possible via community contributions.

Adds DSpark speculative decoding support to llama.cpp via pull request, enhancing inference performance.

MM-ShiftKV is a training-free method that improves KV cache selection for multimodal LLMs by approximating decoding-time query behavior during prefilling, reducing memory footprint while preserving performance.

Introduces Latent Briefing, a method for multi-agent systems to communicate via KV cache compaction, achieving 31% fewer tokens with same accuracy and up to 20x speedup.

Krasis, a MoE-focused runtime, enables running the 397B-parameter Ornith model on a single RTX PRO 6000 Blackwell 96GB GPU with ~20-24 tok/s decode by dynamically managing expert residency in VRAM.

Modified SGLang to support Qwen and Laguna models on V100 GPUs using custom FlashAttention and Marlin kernels, achieving decent throughput on 4xV100 hardware.

Baseten details how it built the fastest API for GLM-5.2, achieving over double the launch-day performance and introducing a latency-optimized Fast version for coding and agents, with further improvements planned.

The author tested the chain of draft paper on their own setup, achieving 74% fewer tokens for the exact same answers.