Serving Deepseek v4 Flash 0731 on 2x DGX Spark — 5-7 GB OS headroom, what would you do to lower VRAM usage and increase OS available RAM?

Hey all, I'm serving DSv4Flash 0731 on a cluster of 2x DGX Sparks but am running into constant issues with having almost no RAM (unified memory) left for the OS/cache and I'd love to hear the community feedback on what I could do to get more RAM for headroom. The DGX has an issue where it locks up when RAM fills, so having more headroom allows things to be comfortably more stable. Question: DeepSeek-V4-Flash-0731 (304B MoE) at full 1M context on 2x NVIDIA DGX Spark — how can we free more RAM for the OS? Hardware 2x DGX Spark (GB10 Grace Blackwell, SM121): 128 GB unified memory each (~121.7 GiB visible), ~273 GB/s bandwidth, 20-core ARM, 4 TB NVMe, DGX OS (headless, GUI/desktop services disabled) Direct-connected ConnectX-7 fabric, dual links, RoCE, MTU 9000; NCCL confirmed using IB transport (~200 Gbps) Model DeepSeek-V4-Flash-0731, native FP8 checkpoint (167 GB safetensors) 304B total params, MoE: 256 routed experts/layer, 6 active + 1 shared per token, 43 layers Sparse attention (indexer top-k 512) + built-in KV compression (4x / 128x per-layer groups), YaRN-calibrated 1M context Serving stack vLLM 0.26.1rc1 built from source (the DeepSeek V4 SM12x support PR), CUDA 13.0 toolchain, running in containers Tensor parallel TP=2 across the two boxes (--nnodes 2, multiprocessing backend, no Ray) DSpark speculative decoding (method "dspark", num_speculative_tokens=5) — ~80% draft acceptance llama-swap as the single OpenAI-compatible endpoint / model router; Open WebUI frontend Key engine flags --kv-cache-dtype fp8_ds_mla (8-bit packed 584 B/token/layer; measured ~3.9 KB/token effective across all layers with the model's compression) --max-model-len 1048576 (full 1M) --gpu-memory-utilization 0.82 ← the knob in question --max-num-batched-tokens 4096 (larger values blow up per-request block accounting via tiny-block-size compressor-state cache groups — each 4-token block pins a ~1 MB slab) --max-num-seqs 6, prefix caching + chunked prefill on, CUDA graphs on tokenizer/tool/reasoning parsers: deepseek_v4 Memory anatomy per node (measured) Weights + activations + CUDA graphs: ~87–89 GiB (immovable; weights are ~83.5 GiB/node under TP=2) KV cache pool: ~11.2 GiB → ~1.6M tokens capacity (1.53x concurrency at 1M) Left for OS + services: ~5–7 GB idle, dips to ~2–3 GB during marathon 1M-context sessions (slow host-side leak, ~100–200 MB/h under sustained long-context load) Watchdogs: earlyoom at absolute 1.5/1.0 GiB floors, custom graceful unloader at 2 GB, vm.min_free_kbytes=2 GiB, swap OFF (on unified memory, swap turns a clean OOM kill into a 30-min livelock) Performance (so you know what we don't want to lose) 82 tok/s decode single-stream, ~1,400 tok/s prefill, 146K-token needle test passes 3/3 What we've already ruled out (please don't suggest these) Lowering gpu-memory-utilization below 0.82: at 1M the KV pool hits vLLM's serve-one-request floor and it refuses to start Lowering max-model-len alone: frees nothing — the pool is sized by the budget %, not by context cgroup/docker memory caps on the model containers: reclaim-loops on file-backed pages and starves sshd (tried, worse than nothing) Enabling swap: livelocks the whole box on unified memory CPU offload / --cpu-offload-gb: meaningless on unified memory (one pool) LMCache disk tier: incompatible — asserts on the packed fp8_ds_mla layout ("kv_cache last dim must be 584, got 512") Weight streaming from NVMe: 40x bandwidth gap, measured ~14 tok/s vs 82 The actual question On identical unified-memory boxes, what host-side or vLLM-side levers actually reduce resident overhead? Specifically curious about: Shrinking vLLM's host-process footprint (API server + engine + worker processes RSS) NCCL buffer/registration tuning for 2-node TP (NCCL_BUFFSIZE etc.) — free GBs or just MBs? torch.compile / inductor / CUDA graph memory: is --enforce-eager's memory savings worth the latency hit on Blackwell, and has anyone measured it for big MoE? Anyone found the source of slow host-RSS growth (~100–200 MB/h) in long-context vLLM serving? Any DGX OS / GB10-specific reservations (driver, carveouts) that can be trimmed safely? A huge thanks in advance for any and all your responses!!