The mismatch
The model: DeepSeek V4 Flash DSpark, the abliterated community variant — 156 GB of FP8 weights, MoE architecture, 1M tokens of native context, designed on the assumption that tensor-parallel traffic crosses NVLink between datacenter GPUs.
The hardware: two RTX PRO 6000 Blackwell Max-Q cards, 96 GB each, on PCIe Gen4 x16 under a single EPYC 7302. No NVLink. Every synchronization in a TP=2 forward pass crosses the PCIe host bridge instead.
Copying the 166 GB checkpoint to the serving node took 33 minutes at a sustained 81 MB/s. That was the easy part.
The wall at 1001 MiB
Every container followed the same script: NCCL initialized cleanly across both GPUs, workers spawned, and weight loading froze at exactly 1001 MiB per card. No error, no timeout, no progress. One fork froze at 891 MiB instead, which at least ruled out coincidence.
dmesg had the answer: AMD-Vi: IO_PAGE_FAULT on GPU 1. The EPYC IOMMU was intercepting the large DMA transfers Blackwell issues during weight loading and blocking them at the hardware level.
NCCL_P2P_DISABLE=1 changed nothing. Runtime IOMMU passthrough failed because the IOMMU group included a PCI bridge. The fix that held was one kernel parameter in GRUB: iommu.passthrough=1, then a reboot. After that, weight loading proceeded.
Twelve runs, one working config
Six vLLM images, twelve iterations. Hopper PTX crashed on SM120. Triton doesn't support SM120 yet. The MARLIN FP4 kernels crashed against the R580 driver.
What finally worked: an SM120 community build with VLLM_TRITON_MLA_SPARSE=1, which routes attention through TileLang and compiles working Blackwell kernels at runtime. First launch pays the JIT cost; CUDA graphs take over after.
VLLM_TRITON_MLA_SPARSE=1(plus its CUDAGRAPH companion flag) — runtime-compiled SM120 kernels--disable-custom-all-reduce— forces NCCL's PCIe-safe all-reduce--kv-cache-dtype fp8— NVFP4 kernel support varies by image
The rest of the recipe: TP=2, a 131,072-token max length, 0.92 GPU memory utilization, block size 256, four concurrent sequences.
What it does now
Recorded results from the July 2026 deployment are shown below. The serving configuration allowed 131,072 tokens. The archived record does not specify benchmark input/output token counts, cache state for each run, or whether the throughput test used the capped proxy. These results should not be read as a verified full-128K-prompt benchmark.
72.1 tok/s
single stream; configured limit 128K
108.8 tok/s
across four concurrent streams
~1.2 s
to first token, warm CUDA graphs
11+ h
of serving with flat memory, zero OOM
88.6 / 96 GB per card
recorded total GPU memory usage
27 s
model load from safetensors
TP=2
over PCIe Gen4 x16
<8 W
idle draw, never throttled at 300 W
Served without exposing the LAN
The endpoint is a tunnel into a FastAPI proxy into vLLM. The proxy enforces Bearer auth — verified 401 without a key, 403 with a wrong one — caps prompts at 10K characters, outputs at 4K tokens, request bodies at 2 MB, and forwards structured tool-calling requests unmodified.
One bug worth recording: the first proxy version silently dropped tools and tool_choice, because the Pydantic request model never declared those fields and model_dump(exclude_none=True) erased what it didn't know about. A validating proxy is only as complete as the schema behind it.
iptables inside the proxy container blocks all RFC 1918 space plus the CGNAT range. The container can reach the internet; it cannot see the LAN.
If you're attempting this
- 01Check
dmesgbefore blaming the container. A weight load frozen at a fixed byte count, plus AMD-Vi page faults, means the IOMMU.iommu.passthrough=1in GRUB is the durable fix. - 02On SM120, runtime-compiled kernels are the path of least pain. PTX built for Hopper won't run, and Triton isn't there yet.
- 03Missing NVLink is survivable. Disable the custom all-reduce and let NCCL handle PCIe.
- 04FP8 KV cache is the compatibility play; NVFP4 support is inconsistent.
- 05Budget time for image roulette. Twelve runs to one working config is a normal ratio on new silicon.
- 06MTP speculative decode is blocked for now: the checkpoint names its heads
mtp.0.*, the SM120 forks expectmtp_block.*. When that lands, throughput should jump an estimated 2 to 3×.
Field conditions
The archived record reports a 156 GB FP8 model, a 166 GB transferred checkpoint and 88.6 GB total usage on each 96 GB GPU. These describe different quantities. The previous 39 GB-per-card weight allocation does not reconcile with the stated checkpoint, so a resident-weight and KV-cache breakdown is withheld until the allocation logs are checked. No offload or precision conversion is inferred from these figures.
The cards run at 300 W Max-Q and have not throttled once; idle draw sits under 8 W.