Auto-fit vs tuned MoE offload: 564 → 1330 pp tok/s, unchanged decode (Qwen3.6-35B-A3B Q6 / RTX 3090)

Reddit r/LocalLLaMA 工具

摘要

A developer benchmarks Qwen3.6-35B-A3B Q6 on an RTX 3090, showing that offloading eight MoE expert layers to CPU and increasing batch sizes improves prompt processing by 2.36× (564→1330 tok/s) with no decode speed regression, using evolutionary search to find the tuning config.

TL;DR: On a Qwen3.6-35B-A3B Q6 setup sized for 64K context on a 24GB RTX 3090, spilling eight MoE expert layers to CPU freed enough VRAM to increase -b from 512 to 1024 and -ub from 128 to 512. Prompt processing improved by 2.36×, while generation speed remained unchanged within measurement noise. Benchmark Auto-fit baseline Tuned Result PP4K 564.5 tok/s 1330.0 tok/s 2.36× TG4K 97.4 tok/s 97.7 tok/s Within noise TG32K 81.6 tok/s 84.0 tok/s Within noise These are PP measurements with a 4K prompt and TG measurements at 4K and 32K context depth. The configurations were sized against a 64K context requirement; this is not a 64K-depth throughput benchmark. I first used auto-fit to establish a feasible configuration. Its resulting batch settings were -b 512 -ub 128; I hard-coded them in the baseline command below so the comparison is reproducible. The tuned configuration deliberately moves eight layers’ MoE expert weights to CPU: -ot 'blk\.(1[2-9])\.ffn_.*_exps\.weight=CPU' \ -b 1024 -ub 512 -ngl 41 This is a joint-configuration result: CPU offload frees VRAM, and the larger batch/micro-batch uses that memory to accelerate prefill. It is not an isolated claim that CPU offload alone improves performance. Full reproduction Baseline, reproduces the auto-fit configuration: llama-bench \ -m Qwen3.6-35B-A3B-UD-Q6_K.gguf \ -fitt 1024 -fitc 65536 \ -t 7 -b 512 -ub 128 \ -fa on -ctk q8_0 -ctv q8_0 -mmp 1 \ -p 4096 -n 64 -r 2 \ -d 4096,32768 Tuned: llama-bench \ -m Qwen3.6-35B-A3B-UD-Q6_K.gguf \ -t 7 -b 1024 -ub 512 -ngl 41 \ -fa on -ctk q8_0 -ctv q8_0 -mmp 1 \ -ot 'blk\.(1[2-9])\.ffn_.*_exps\.weight=CPU' \ -p 4096 -n 64 -r 2 \ -d 4096,32768 Environment Model: unsloth/Qwen3.6-35B-A3B-GGUF Quant: Qwen3.6-35B-A3B-UD-Q6_K.gguf, 27.3 GiB SHA-256: 4fe53b148b46f9b88830e2a3055c5b15c3a4d1e3ddc9a1384a108d8b9d59f043 GPU: RTX 3090, 24 GiB CPU: Threadripper PRO 3955WX, with seven cores available to the rental RAM: approximately 100 GB DDR4 llama.cpp: commit 571d0d5 Build: -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON, Release Peak tuned VRAM: 23,468 / 24,576 MiB, leaving approximately 1.1 GiB Caveat: these are two-repetition measurements, with approximately 1.6% observed run-to-run drift. Treat the TG differences as noise; the meaningful result is the 2.36× PP improvement without an observed decode regression. Method I used evolutionary search to get to above config (LEVI), which I ran for roughly 100 evaluations / 40 minutes (https://github.com/ttanv/levi). For now I'm only evolving basic flags and configs, but I'm really looking forward to more unconventional edits, perhaps editing parts of llama cpp. The goal is to rewrite whatever part of the stack that is generic enough to leave bespoke optimizations on the table, so the serving engine is fully custom to the model+hardware combo. Faithful and fast evals are hard tho :( . If any of you have suggestions or experience on this, would love to hear. I also want to test whether this generalizes and can be useful in other setups. If you have a partially offloaded MoE or another near-VRAM-limit setup, reply with: GPU CPU and RAM configuration Exact GGUF Target context length Current command Whether you care most about PP, TG, or fitting a larger model I want to try genuinely different setups and see how it generalizes. I'm looking for especially more niche and custome type of setups. Tho hopefully something not too large lol, since I'm relying on vast ai for this.
查看原文

相似文章

4x RTX 3090 上的 Qwen3.5-27B、Qwen3.5-122B 和 Qwen3.6-35B —— MoE 模型在严格全局规则下的表现困境

Reddit r/LocalLLaMA

潜水多年的老用户,首次发帖。在 4 张 RTX 3090 上对三款 Qwen 模型分别进行了 20 多个会话的实时智能体工作测试——**Qwen3.5-27B** 稠密模型、**Qwen3.5-122B-A10B** MoE 和 **Qwen3.6-35B-A3B** MoE。以下数据均解析自持续真实负载下的 vLLM 日志,而非合成基准测试。**本文所有数据的关键负载背景:** 测试框架是一个多智能体编排器,同时运行 1-6 个并发的 OpenCode 会话,Prompt 长度为 30-60k token,并且强制执行**严格的 Bash 允许列表