156 GB of DeepSeek on two GPUs.156 GB of DeepSeek on two GPUs.

The model: DeepSeek V4 Flash DSpark, the abliterated community variant — 156 GB of FP8 weights, MoE architecture, 1M tokens of native context, designed on the assumption that tensor-parallel traffic crosses NVLink between datacenter GPUs.

The hardware: two RTX PRO 6000 Blackwell Max-Q cards, 96 GB each, on PCIe Gen4 x16 under a single EPYC 7302. No NVLink. Every synchronization in a TP=2 forward pass crosses the PCIe host bridge instead.

Copying the 166 GB checkpoint to the serving node took 33 minutes at a sustained 81 MB/s. That was the easy part.

Every container followed the same script: NCCL initialized cleanly across both GPUs, workers spawned, and weight loading froze at exactly 1001 MiB per card. No error, no timeout, no progress. One fork froze at 891 MiB instead, which at least ruled out coincidence.

dmesg had the answer: AMD-Vi: IO_PAGE_FAULT on GPU 1. The EPYC IOMMU was intercepting the large DMA transfers Blackwell issues during weight loading and blocking them at the hardware level.

NCCL_P2P_DISABLE=1 changed nothing. Runtime IOMMU passthrough failed because the IOMMU group included a PCI bridge. The fix that held was one kernel parameter in GRUB: iommu.passthrough=1, then a reboot. After that, weight loading proceeded.

Six vLLM images, twelve iterations. Hopper PTX crashed on SM120. Triton doesn't support SM120 yet. The MARLIN FP4 kernels crashed against the R580 driver.

What finally worked: an SM120 community build with VLLM_TRITON_MLA_SPARSE=1, which routes attention through TileLang and compiles working Blackwell kernels at runtime. First launch pays the JIT cost; CUDA graphs take over after.

• VLLM_TRITON_MLA_SPARSE=1 (plus its CUDAGRAPH companion flag) — runtime-compiled SM120 kernels
• --disable-custom-all-reduce — forces NCCL's PCIe-safe all-reduce
• --kv-cache-dtype fp8 — NVFP4 kernel support varies by image

The rest of the recipe: TP=2, a 131,072-token max length, 0.92 GPU memory utilization, block size 256, four concurrent sequences.

Sustained numbers from the production deployment, measured end to end:

The endpoint is a tunnel into a FastAPI proxy into vLLM. The proxy enforces Bearer auth — verified 401 without a key, 403 with a wrong one — caps prompts at 10K characters, outputs at 4K tokens, request bodies at 2 MB, and forwards structured tool-calling requests unmodified.

One bug worth recording: the first proxy version silently dropped tools and tool_choice, because the Pydantic request model never declared those fields and model_dump(exclude_none=True) erased what it didn't know about. A validating proxy is only as complete as the schema behind it.

iptables inside the proxy container blocks all RFC 1918 space plus the CGNAT range. The container can reach the internet; it cannot see the LAN.

1. 01Check dmesg before blaming the container. A weight load frozen at a fixed byte count, plus AMD-Vi page faults, means the IOMMU. iommu.passthrough=1 in GRUB is the durable fix.
2. 02On SM120, runtime-compiled kernels are the path of least pain. PTX built for Hopper won't run, and Triton isn't there yet.
3. 03Missing NVLink is survivable. Disable the custom all-reduce and let NCCL handle PCIe.
4. 04FP8 KV cache is the compatibility play; NVFP4 support is inconsistent.
5. 05Budget time for image roulette. Twelve runs to one working config is a normal ratio on new silicon.
6. 06MTP speculative decode is blocked for now: the checkpoint names its heads mtp.0.*, the SM120 forks expect mtp_block.*. When that lands, throughput should jump an estimated 2 to 3×.

FP8 stores each weight in 8 bits instead of 16 — half the byte count of a BF16 checkpoint of the same parameter count, which is what let 39 GB of weights and roughly 8 GB of KV cache per card fit inside 96 GB alongside a 128K-token context.

The cards run at 300 W Max-Q and have not throttled once; idle draw sits under 8 W.