NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B on 2x3090s

I managed to get this model working on 2x 3090s with full 262k ctx and N=4, if anyone is interested to try it, thanks to this quant: https://huggingface.co/danielrmay/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-W4A16 Nemotron-Labs-3-Puzzle-75B-A9B (W4A16) on 2× RTX 3090 — vLLM, no CPU offload, full 262K, N=4 Hardware: 2× RTX 3090 (SM 8.6), PCIe Gen4 ×8, no NVLink, aikitoria BAR1-P2P patched driver active. Engine: vllm/vllm-openai:cu129-nightly dev1060 (9e57de71). Model: danielrmay/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-W4A16 (https://huggingface.co/danielrmay/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-W4A16) (compressed-tensors: INT4 experts / INT8 shared+mamba / BF16 attn+latent+router+embed+lm_head, MTP stripped, 41.48 GiB). docker run -d --name vllm-puzzle --gpus '"device=0,1"' --ipc=host --network=host --ulimit memlock=-1 --ulimit stack=67108864 -e CUDA_DEVICE_ORDER=PCI_BUS_ID -e VLLM_NO_USAGE_STATS=1 -e VLLM_USE_FLASHINFER_SAMPLER=0 -e VLLM_ATTENTION_BACKEND=TRITON_ATTN -e VLLM_SKIP_P2P_CHECK=1 -e PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.6,max_split_size_mb:128 -v /path/to/puzzle-w4a16:/model:ro --entrypoint vllm vllm/vllm-openai:cu129-nightly-x86_64 serve /model --served-model-name puzzle-w4a16 --host 0.0.0.0 --port 8000 --trust-remote-code --tensor-parallel-size 2 --enable-expert-parallel --kv-cache-dtype int8_per_token_head --mamba-backend triton --kv-cache-memory-bytes 680000000 --gpu-memory-utilization 0.965 --max-num-seqs 4 --max-num-batched-tokens 1024 --max-model-len 262144 --compilation-config '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,2,4]}' --reasoning-parser nemotron_v3 --tool-call-parser qwen3_coder --enable-auto-tool-choice PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.6,max_split_size_mb:128 — the breakthrough. Reclaims a ~1.07 GiB reserved-but-unallocated fragmentation that expandable_segments:True could not, and rebalances the two ranks. Prerequisite for everything else. (Also makes custom-all-reduce IPC handles work, unlike expandable_segments.) PIECEWISE CUDA graphs — biggest decode lever (27.9 eager → 88+ tok/s). Eager Python dispatch over the 88-layer hybrid was the real N=1 bottleneck, not TP all-reduce. splitting_ops keeps mamba/attention eager; only MoE/MLP subgraphs are captured (~tens of MiB). FULL graphs do NOT fit real 24 GB — PIECEWISE does. Capture sizes must cover max-num-seqs ([1,2,4] for N=4) or N≥3 decode silently degrades to eager. custom-all-reduce ON (default, via aikitoria BAR1 P2P + the GC allocator): +8% N=1 / +6% N=4. Note it accelerates only the TP all-reduce (8 attn + 40 mamba layers) — the EP MoE all-to-all stays PyNCCL. Measured (real 3090s, not simulated): N=1 decode 93.8 tok/s, prefill ~3660 tok/s u/4-8K N=4 aggregate 255 tok/s (per-stream 69), TTFT p50 0.61s, 0 preempt/0 OOM Full 262K context (int8 KV pool 278K tokens), GPU floor 267/287 MiB 52K needle recall PASS; 847×293=248171, "all but 9"=9, clean qwen3_coder tool call