Tag
A developer released cudamemcpy-tuner, an open-source tool that speeds up pageable cudaMemcpy transfers up to 5x on NVIDIA B300 GPUs, boosting 1 GB CPU-to-GPU copies from 10.2 GB/s to 51.5 GB/s via a `cmt tune` command.
Strata's latest update significantly boosts inference performance for AMD R9700 users running Flash Next, roughly doubling token output from ~70 tok/s to 140-160 tok/s while still trailing Radiance. The user notes quantization limits (Q5 falling just short of VRAM for split-engine mode) and hopes future updates close the gap further.
A technical analysis of optimizing LLM inference using agent fleets, based on DeepSeek 4.1 Flash optimization reaching 2300 tok/s, highlighting that most gains come from configs/graphs rather than new kernels, questioning megakernel approaches, and emphasizing verification and convergence in automated optimization runs.
A detailed comparison of inference engines running Qwen3.8-27B and Flash-Next on consumer hardware, with verified decode/prefill numbers and cost-per-euro analysis showing two mid-range GPUs can beat a single RTX 5090 by 3–10× per euro, while DDR5 RAM now makes up ~40% of experts-in-RAM builds.
一篇技术博客深入解释了投机解码为什么快:并非因为总FLOPs减少(实际上反而增加),而是因为它改变了GPU的工作类型——在低批量场景下把原本空闲的算力从内存瓶颈(memory-bound)转为计算瓶颈(compute-bound),用单序列的'深度'换取了批量解码的'广度'。
User benchmarked Gemma 4 12B IT (GGUF) on a Radeon RX 9070 XT GPU, reaching about 85.3 tokens per second via localmaxxing.com, noting solid performance with some rough edges remaining.
LlamAmpere v0.4 is released with Ampere-specific improvements, achieving over 95 tokens per second and supporting 262K context for the Qwen3.8-27B model on a single NVIDIA 3090 GPU.
Modal and Carnegie Mellon University's Full Stack Data Lab developed Quail, an inference engine that integrates query planning with inference to achieve over a billion tokens processed per minute on a single H100 GPU, outperforming vLLM by 10x in specific scenarios.
Achieved 85.6 tokens per second using the Qwen3.8:27b model on a single RTX 5090 GPU.
Tests of Qwen-Image-2.1 on an NVIDIA 5070 Ti GPU demonstrate it fits within 16GB VRAM, with generation times around 25s for 1024² images, and highlight excellent prompt adherence and typography capabilities.
A draft model for Qwen 3.8 27B is highly effective on 16 GB GPUs, achieving around 60 tokens per second on an RX 9070 XT and offering better VRAM efficiency than built-in MTP.
Showcases the performance of the Qwen3.8-27B AI model running on a 12GB RTX 3080 Ti GPU, achieving 47 tokens per second at 128K context using llama.cpp with specific quantization settings.
A user reports achieving high token throughput with the Ninfer tool on an NVIDIA RTX 5090 GPU using a Qwen 3.8B model, significantly outperforming llama.cpp.
A benchmark comparison of community quantized Qwen 3.8 27B models on RTX 6000 GPUs versus Claude Opus 4.6, evaluating token generation speed and efficiency for creating HTML games.
LLM4LLM introduces a deployment-aware closed-loop optimization framework to bridge kernel benchmarks and real LLM inference, achieving up to 6.98x speedups on H100 GPUs.
The article describes hosting the Kimi K3 AI model with 2.8 trillion parameters using 8 B300 GPUs, achieving 92 tokens per second and costing $190 per million tokens, while comparing it with Unsloth's dynamic GGUF quantization method.
UC Berkeley has open-sourced FreeToken, a tool that significantly speeds up local AI inference on consumer GPUs, achieving up to 39.3 tok/s on an 8GB RTX 4060 laptop.
A user tested the Qwen3.8 27B model on SVG generation with a complex prompt and found the results exceeded expectations, highlighting the model's capability in creative tasks.
The author optimized the Qwen3.8-27B model inference on an RTX 3090 GPU, achieving up to 99 tokens per second for single requests and 1150 tps with batch processing through various quantization and optimization techniques, and released the updated code on GitHub.
The article explains how to determine if a GPU workload is compute-bound or memory-bound by analyzing operations per byte fetched from HBM, using NVIDIA's H100 as an example, and discusses how batching and prompt length affect performance.