Qwen3.8-Flash-Next 在单张96 GB显卡上支持170K上下文窗口，速度约110 tokens/秒。

我将量化后的 n-gram 模型转为 INT4 格式，其占用 32GB 空间，通过内存映射从磁盘加载。经测试，在 150-160k 上下文长度下运行效果极佳，且我在执行单线程长序列任务时持续监控显存使用情况——剩余显存完全足以将上下文长度推至 170k 以上。模型质量确实出色！它成功生成了几个复杂的 HTML 游戏，甚至摸索出无需浏览器就能运行游戏的方法（我的 Ubuntu 系统没有图形界面），并且持续优化改进……以下是操作步骤： ```bash # 下载基础模型（排除大文件） hf download primitive-ai/Qwen3.8-Flash-Next-NVFP4 \ --exclude "ple-bf16-*" \ --local-dir ./flash-next cd flash-next # 下载量化后的 PLE 组件 hf download primitive-ai/Qwen3.8-Flash-Next-PLE-quant \ --include "ples_int4/*" \ --local-dir . # 下载量化处理脚本 hf download primitive-ai/Qwen3.8-Flash-Next-PLE-quant \ worker_image_quant.py ple_layer_quant.py \ --local-dir . ``` 跳过 ple-bf16-* 文件（可节省 100GB 空间，但需修正索引）。修正步骤如下： ```python import json p = 'model.safetensors.index.json' d = json.load(open(p)) wm = d['weight_map'] # 找出所有指向 ple-bf16 的条目 drop = [k for k, v in wm.items() if v.startswith('ple-bf16-')] assert len(drop) == 128 and all('ngram_embedding' in k for k in drop) # 从权重映射中删除 for k in drop: del wm[k] json.dump(d, open(p, 'w')) ``` 我本意是将 n-gram 数据存入 64GB 内存，但最终 n-gram 模型、专家模块与 KV 缓存全部驻留在 GPU 显存中，运行速度极快！单流处理速度达 76-125 token/秒。在代码和 JSON 任务上多令牌预测接受率约 87%，纯对话场景约 40%。长序列任务前缀缓存命中率超 90%。单张 GPU 显存占用约 89GB，页面缓存约 33GB，支持 165-170K 上下文长度。 以下是在我的单节点 k0s 集群（单卡 Pro 6000 显卡）上部署的完整配置（CUDA 13/580，Ubuntu 24.04 无图形界面）： ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: vllm-qwen38-flash-next namespace: default spec: replicas: 1 strategy: type: Recreate # 确保单GPU独占 selector: matchLabels: app: vllm-qwen38-flash-next template: metadata: labels: app: vllm-qwen38-flash-next spec: runtimeClassName: nvidia nodeSelector: nvidia.com/gpu.present: "true" tolerations: - effect: NoSchedule key: nvidia.com/gpu operator: Exists initContainers: - name: init-echo image: busybox:1.36 command: ["/bin/sh", "-c"] args: ['echo "I am here" > /opt/reservation/echo.txt'] volumeMounts: - mountPath: /opt/reservation name: reservation-volume containers: - name: vllm-server image: vllm/vllm-openai:qwen38-flash-next imagePullPolicy: IfNotPresent args: - --model - /model # ---- 单卡PLE卸载关键配置 ---- - --distributed-executor-backend - mp # ----------------------------- - --dtype - auto - --kv-cache-dtype - auto - --gpu-memory-utilization - "0.95" - --max-model-len - "173400" - --tensor-parallel-size - "1" - --pipeline-parallel-size - "1" - --limit-mm-per-prompt - '{"image":12,"video":2}' - --max-num-batched-tokens - "16384" - --max-num-seqs - "4" - --enable-chunked-prefill - --enable-prefix-caching - --no-enable-flashinfer-autotune - --speculative-config - '{"method":"mtp","num_speculative_tokens":3}' - --override-generation-config - '{"temperature":1,"top_p":0.95,"top_k":20}' - --enable-auto-tool-choice - --reasoning-parser - qwen3 - --tool-call-parser - qwen3_coder - --trust-remote-code - --api-key - key1 - --host - 0.0.0.0 - --port - "8990" - --served-model-name - qwen38-flash env: - name: VLLM_PLE_CPU_OFFLOAD value: "1" - name: VLLM_PLE_OFFLOAD_READY_TIMEOUT value: "1800" # 确认：worker_image_quant.py:419 读取此变量 # 指向INT4表目录；覆盖层进行内存映射（MADV_RANDOM，模式"c"） - name: VLLM_PLE_QUANT_DIR value: /model/ples_int4 # 刻意不设置 VLLM_PLE_DISK_OFFLOAD_DIR（第450行） # 那是启用NVMe上的BF16表路径 - name: VLLM_LOGGING_LEVEL value: INFO - name: OMP_NUM_THREADS value: "1" - name: PYTORCH_CUDA_ALLOC_CONF value: max_split_size_mb:512 ports: - containerPort: 8990 protocol: TCP resources: limits: cpu: "12" nvidia.com/gpu: "1" requests: cpu: "8" nvidia.com/gpu: "1" securityContext: capabilities: add: ["IPC_LOCK", "SYS_ADMIN"] startupProbe: httpGet: path: /health port: 8990 periodSeconds: 15 failureThreshold: 80 # 约20分钟；首次启动需加载查找表 readinessProbe: httpGet: path: /health port: 8990 periodSeconds: 20 failureThreshold: 3 lifecycle: preStop: exec: command: ["/bin/sh", "-c", "rm -f /opt/reservation/echo.txt"] volumeMounts: - mountPath: /model name: model-volume readOnly: true # --- 双文件量化PLE覆盖层 --- - mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/ple_offload/worker.py name: ple-worker-overlay readOnly: true - mountPath: /usr/local/lib/python3.12/dist-packages/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py name: ple-layer-overlay readOnly: true - mountPath: /ples_int4 name: ple-tables readOnly: true # ---------------------------- - mountPath: /dev/shm name: dshm - mountPath: /root/.cache/vllm name: vllm-cache - mountPath: /root/.triton name: triton-cache - mountPath: /opt/reservation name: reservation-volume volumes: - name: model-volume hostPath: path: /directory/models/Qwen3.8-Flash-Next-NVFP4 type: Directory - name: ple-worker-overlay hostPath: path: /directory/ple-overlay/worker_image_quant.py type: File - name: ple-tables hostPath: path: /directory/models/Qwen3.8-Flash-Next-NVFP4/ples_int4 type: Directory - name: ple-layer-overlay hostPath: path: /directory/ple-overlay/ple_layer_quant.py type: File - name: dshm emptyDir: medium: Memory sizeLimit: 32Gi - name: reservation-volume hostPath: path: /opt/reservation type: DirectoryOrCreate - name: vllm-cache hostPath: path: /var/cache/vllm type: DirectoryOrCreate - name: triton-cache hostPath: path: /var/cache/triton type: DirectoryOrCreate --- apiVersion: v1 kind: Service metadata: name: vllm-qwen38-flash-next namespace: default spec: type: NodePort selector: app: vllm-qwen38-flash-next ports: - name: http port: 8990 targetPort: 8990 nodePort: 32001 protocol: TCP ``` 衷心感谢 primitive-ai（无论您是哪位开发者）。