Tag
Cerebras co-founder and CEO Andrew Feldman explains why the company's wafer-scale architecture claims ~2,500× faster memory-to-compute movement than GPUs during LLM decode, since Cerebras keeps model weights in on-chip SRAM rather than HBM.
PyTorch's Helion kernel DSL is integrated into vLLM's linear backend, where a single autotuned GEMM implementation covers Standard GEMM, Split-K, and Swap-AB variants, outperforming CUTLASS and DeepGEMM backends on NVIDIA Hopper GPUs with over 10% throughput gains on some workloads.
A step-by-step, first-principles explainer on how prefill and decode actually work inside LLM inference, starting from how ChatGPT-style chats process prompts via tokenization and autoregressive token generation.
A user shares and asks for feedback on running Qwen3.8-Flash-Next in EXL3 5.05bpw on a single Radeon RX 9700 (gfx12) with mixed GPU/CPU expert offloading, reporting ~863 t/s prefill and ~35 t/s decode at 230k context.
作者对开源 LLM 推理服务 Gufo 0.4.0 进行实测复现：其宣传的 70 tok/s 确实存在，但主要来自重复词 prompt 下的投机解码命中率，普通 prompt 上约 39 tok/s；与 halogen 相比，日常生成速度慢约 13-18%，但长 prompt 处理更快。
Janus 是一个单文件 Go 二进制工具，通过 Vulkan 在 AMD/Intel/NVIDIA GPU 或 CPU 上运行 GGUF 模型，并提供 OpenAI 兼容 API，无需 Python、Docker 或 Ollama。
作者发布 Slipstream——一款针对 Apple Silicon 的 C++ Metal 推理引擎，支持原生 SSD 专家流式加载与投机草稿解码，可在 64GB Mac 上以 41–52 tok/s 运行 95.5 GiB 的 Qwen3.8-Flash-Next 模型，比 llama.cpp 快 1.76 倍，且 130k 长上下文下速度不衰减。
Edge0's application layer has been open-sourced, enabling users to run the Edge0 35B language model (1–2.5 GB peak memory) locally on iOS, macOS, Android, and Windows devices.
A user reports solid results running a $5400 eBay 8x V100 server for local LLM inference, using nvfp4 NVIDIA checkpoints via a heavily optimized vLLM fork (1CatAI/1Cat-vLLM) that unpacks to fp16 on the fly, achieving over 200 tok/s on a 27B model with TP=4 and strong prefill throughput.
An in-depth explainer on how LLM inference really works: the prefill/decode loop, KV caching, how file encodings and quantization labels (e.g. NVFP4) determine which kernels a GPU actually runs, demonstrated with Qwen 3.8 27B on an RTX 5090 — same model, same label, two very different kernel paths and performance.
An educational walkthrough of the LLM serving scheduling loop, explaining how inference requests move through waiting queues, batch construction, prefill, decode, and continuous batching to keep GPUs saturated as requests finish at different times.
DEdit introduces a diffusion-based drafter for speculative decoding that iteratively edits drafts via token-to-token predictions, using a ProposalMix training scheme to repair errors while preserving correct tokens. It achieves macro-average speedups of 5.72× and 5.97× on Qwen3-4B and Qwen3-8B across seven benchmarks.
Vercel AI Gateway的CEO @rauchg 表示，大多数token聚合器充斥着噪音，包括那些要求你把数据用于训练的token促销，以及声称支持ZDR（零数据保留）但信誉存疑的提供商；而Vercel AI Gateway凭借真实客户用量、40万+付费客户规模和零加价，成为理解全球AI token流向的最可信数据源。
Eric Zhang shares a steady-state analysis arguing that PD disaggregation mainly improves mean per-token latency for prefill-heavy workloads (giving ~1/(decode-time fraction) interactivity gains) and is largely unnecessary for decode-bound, low-latency inference — validated in production, and echoed by Horace He's remark that few have really thought through PD disagg vs. mixed-chunk prefill.
AMD's ROCm/FastFlowLM now supports running Qwen 3.8 27B (and other MoE, vision, audio, and embedding models) on Ryzen AI XDNA2 NPUs with no GPU required, offering up to 256k context in a lightweight 17 MB runtime — though decode speed on the 27B model is noted as a sluggish 1 token/s.
This article argues the AI race narrative has shifted: Western labs now quietly adopt open Chinese advances like DeepSeek's KV cache compression (437x smaller than V1), which has driven inference and cached-token pricing at Anthropic and OpenAI sharply down.
A Hacker News discussion exploring what API/interface should be used to build general real-time asynchronous LLM agents that can react to new stimuli (interruptions, live event streams) mid-inference, referencing the AsyncLLM preprint and existing vendor real-time APIs as partial solutions.
User @loktar00 shares that RTX 3090s can be run at much lower power limits (down to ~120W) than the usual 210-240W range, saving significant heat and power during inference workloads.
A pull request to ggml-org/llama.cpp adds inference support for the GLM-5.3-Flash (GLM5-Next) model, extending the library's coverage of Zhipu AI's GLM family.
The author describes their agent workflow using asyncio.gather to batch vLLM API calls with shared prefixes and asks for best practices to optimize this process, proposing a pre-caching strategy for efficient decoding.