LLMRequirements.com
All news

NVFP4 + Qwen 3.6 35B-A3B on DGX Spark: 97 tok/s single, 322 tok/s at 8 concurrent (decode-only)

By ·

#software#performance#nvidia#dgx-spark#nvfp4

NVIDIA's official NVFP4 checkpoint of Qwen 3.6 35B-A3B (May 28). On a DGX Spark running the technigmaai/dgx-spark recipe we measured 97 tok/s single-stream and 322 tok/s aggregate at 8 concurrent with prefix-cached prefill (42 tok/s per stream). Community MTP-3 sweet spot adds another lift on FP8.

NVIDIA uploaded an official nvidia/Qwen3.6-35B-A3B-NVFP4 checkpoint on 2026-05-28 (HF card). On a DGX Spark, with vLLM, FlashInfer attention, the CUTLASS-FP4 MoE backend, and MTP-3 speculative decoding all wired up by the technigmaai/dgx-spark recipe, the 35B-A3B MoE — already the local-coding favourite at sub-200B — runs at frontier-API decode speeds.

We benchmarked it directly against an in-house Spark running the recipe with --max-num-seqs 8 (raised from the recipe default of 2), num_speculative_tokens: 3 (MTP-3), --enable-prefix-caching, --enable-chunked-prefill, --moe_backend auto with VLLM_USE_FLASHINFER_MOE_FP4=0 (i.e. CUTLASS-FP4 MoE rather than FlashInfer’s FP4 path), --attention-backend flashinfer, --kv-cache-dtype fp8.

Single-stream — 97 tok/s

Three runs at concurrency=1, max_tokens=400, balanced temperature:

RunOutput tokensWall timeOutput tok/s
14004.10 s97.5
24004.02 s99.6
34004.21 s95.1
Mean4004.11 s97.4

This is the number to plan around for agentic-loop workloads — one request at a time, full GPU dedicated to it.

8 concurrent streams (decode-only) — 322 tok/s aggregate, 42 tok/s per stream

To isolate pure decode throughput, we sent the same prompt 8× in parallel (warming the vLLM prefix cache first so prefill is effectively free) and measured per-stream first_token → last_token windows via streamed SSE. Token counts come from vLLM’s usage block at end-of-stream, not chunk counting — important because MTP-3 packs multiple tokens into each SSE event. The server was configured with --max-num-seqs 8 (raised from the recipe’s default of 2), so all 8 streams genuinely shared one 8-deep batch on the GPU rather than serializing through a smaller cap.

MetricValue
Wall time, all 8 finished10.24 s
Total output tokens3,200 (400 × 8)
Aggregate decode throughput322.05 tok/s
Mean per-stream decode42.16 tok/s
Per-stream range40.4 – 44.4 tok/s
Mean TTFT (warm prefix cache)309 ms

The 8 streams clustered tight (4 tok/s spread), so this isn’t head-of-line blocking. Aggregate at concurrency 8 = 3.31× single-stream, with each individual request decoding at ~43 % of single-stream speed.

What the community thread is saying

The canonical Qwen 3.6 35B-A3B discussion on the NVIDIA developer forums (forums.developer.nvidia.com/t/qwen-qwen3-6-35b-a3b-and-fp8-has-landed/366822) has converged on a few stable findings on DGX Spark:

  • MTP-3 is the sweet spot for FP8 (timhbl, 2026-04-16): FP8 baseline 51.2 → MTP-2 58.6 → MTP-3 63.9 tok/s → MTP-4 drops back to 52.9. Acceptance length 2.77 stable across loads.
  • Concurrency scales further than 8 (timhbl): same setup at c=32 → 411.6 tok/s aggregate — i.e., the per-stream cost continues to drop as concurrency rises, until VRAM and KV cache budgets bite.
  • NVFP4 vs FP8 single-stream (Digital_David, 2026-04-16): NVFP4 mmangkad variant ~35 t/s vs FP8 baseline ~52 t/s. The official NVIDIA NVFP4 checkpoint (May 28) closes most of that gap with proper MTP-2 + FlashInfer + Marlin MoE — the 97 t/s we measured beats both numbers from April.
  • TurboQuant Hybrid (grindstone): on the same Spark, swapping FP8 KV cache for hybrid quant lifts KV capacity from ~400 K → 1,530,704 tokens — relevant if you want 1 M-context with the model.
  • Z-Lab’s DFlash variant (say3): different SKU (z-lab/Qwen3.6-35B-A3B-DFlash), reaches 101.7 tok/s on JSON output single-stream and 184.9 t/s aggregate at concurrency 4 — useful comparison point.

What NVFP4 actually is

NVFP4 is NVIDIA’s 4-bit floating-point format (E2M1: one sign, two exponent, one mantissa bit). Each 16-value micro-block carries an E4M3 FP8 scale; a per-tensor FP32 macro scale sits above. Effective storage is ~4.5 bits/value — about 3.5× smaller than FP16, 1.8× smaller than FP8. NVIDIA’s launch post claims <1 % accuracy degradation vs FP8 on DeepSeek-R1-0528 (NVIDIA developer blog), with AIME-2024 scores actually improving 2 % on the 4-bit variant.

The format requires 5th-gen Tensor Cores, so Blackwell-only: GB10 (DGX Spark / OEM clones), RTX PRO 6000 Blackwell, B100/B200/GB200, and the Blackwell-derivative Jetson Thor. Hopper does not get it.

NVFP4 itself shipped in June 2025 — what’s new in May 2026 is the official nvidia/* checkpoint family on Hugging Face. The Qwen 3.6 35B-A3B card joined a wave of NVIDIA-quantized open-weights checkpoints (GLM-5.1, DeepSeek-V4-Pro, Qwen 3.5 122B-A10B). The card claims a 3.06× size reduction vs BF16 — the model lands on disk at ~19 GB.

Why it matters for DGX Spark / GB10 owners

The 35B-A3B MoE is the agentic-coding favourite among sub-200B open-weights models — top of the Hacker News launch thread had Simon Willison reporting it draws better pelicans-on-bicycles than Claude Opus 4.7. The community settled on FP8 via vLLM as the daily-driver path on Spark almost immediately — the thread itself is titled “Qwen/Qwen3.6-35B-A3B (and FP8) has landed!” — and FP8 baseline decode landed in the ~52 tok/s range (Digital_David, post #28). With MTP-3 tuned in (timhbl, post #27), FP8 climbed to 63.9 tok/s.

NVFP4 is the next step on the same vLLM path:

  • ~19 GB on disk instead of ~38 GB FP8 — frees 19 GB of the 128 GB pool for context, KV cache, or a second model.
  • Same vLLM stack — FlashInfer attention, Marlin MoE kernels, MTP-2 speculative decoding native to the recipe; no migration cost.
  • 97 tok/s single-stream vs ~52 t/s FP8 baseline / 63.9 t/s FP8+MTP-3 — i.e. ~1.5× FP8+MTP-3, ~1.9× FP8 baseline.
  • 322 tok/s at 8 concurrent / 411 tok/s at c=32 — vs FP8 community numbers of similar shape.

Steve Scargall’s earlier vLLM-on-Spark recipe (blog, Apr 25) reported 55.9 tok/s single-stream with a less-tuned pipeline. Six weeks of vLLM + NVFP4 + FlashInfer + MTP tuning + a stable Docker recipe (technigmaai/dgx-spark) have lifted single-stream by ~75 %.

How to enable

The cleanest path today is the technigmaai/dgx-spark/spark-vllm-docker/nvidia-Qwen3.6-35B-A3B-NVFP4 recipe (file qwen3.6-35b-a3b-nvfp4-nvidia-v2.yaml). Key flags it sets on vllm serve:

--quantization compressed-tensors    # NVFP4 weights ship as compressed-tensors
--kv-cache-dtype fp8                  # FP8 KV cache, not FP4
--attention-backend flashinfer
--moe_backend auto                    # resolves to CUTLASS-FP4 on Blackwell
--load-format instanttensor           # NVIDIA fast-load format
--max-model-len 262144
--max-num-seqs 2                      # recipe default; bump to 8+ for real batched serving (we ran with 8)
--max-num-batched-tokens 32768
--gpu-memory-utilization 0.7
--enable-prefix-caching
--enable-chunked-prefill
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
--optimization-level 3 --performance-mode throughput

Notable env vars from the same recipe: VLLM_USE_FLASHINFER_MOE_FP4=0 (explicitly opts out of FlashInfer’s FP4 MoE in favour of CUTLASS-FP4), VLLM_MARLIN_USE_ATOMIC_ADD=1, VLLM_FP8_MOE_BACKEND=flashinfer_cutlass (fallback path if you re-quantize to FP8), and CUTE_DSL_ARCH=sm_121a for Blackwell-specific CuTe kernels.

Bump --max-num-seqs if you want the c=32 aggregate ceiling. SGLang support is on NVIDIA’s roadmap but not yet shipped; TRT-LLM has it today for users on that stack.

Caveats

  • Blackwell-only. Hopper H100/H200 owners cannot use NVFP4 — they still need FP8 or Q4_K_M paths.
  • Long-context prefill is still the bottleneck. Single-stream decode at 97 tok/s is excellent; the same Spark hits ~5,600 tok/s prefill on an 11 K-token prompt — fast enough for chat, slow enough to notice on 100 K-token agentic workflows. Long-context TTFT sits in the multi-second range, not sub-second like an API.
  • The 322 tok/s number is decode-only with a warm prefix cache. Real-world batched serving where each user sends distinct prompts will see ~10-15 % lower aggregate because every prompt pays its own prefill. We measured ~290 tok/s with distinct prompts at the same 8-concurrent shape.
  • Per-stream throughput drops fast under concurrency. 1 user → 97 tok/s; 8 users → 42 tok/s each. Plan capacity by your concurrency budget, not by the aggregate number.

Sources