8× RTX Pro 6000 Blackwell server (768 GB) kimi-k2-5-1t-moe @ INT4 (FP8 KV, DCP=8) on vLLM batch ~900 @ 40K (100-conc. aggregate) — local-inference-lab/rtx6kpro wiki (Kimi K2.5 high concurrency, Festr) ↗ I run this Be the first to run this Dual RTX Pro 6000 Blackwell build qwen3-6-27b-dense @ FP8 (fp8 KV, MTP spec=3) on vLLM batch ~894 @ 175K (32-conc. aggregate) — theogravity/dual-rtx-6000-blackwell-qwen3.6-27b-fp8 (coding sweep, seqs=32) ↗ I run this Be the first to run this 8× H100 80 GB server deepseek-v3-671b-moe @ FP8 (TP=8) on vLLM batch ~620 @ 1.024K (100-conc. aggregate) — dzhsurf/deepseek-v3-r1-deploy-and-benchmarks (8xH100 vLLM TP=8, ~100 concurrency) ↗ I run this Be the first to run this 2× Strix Halo cluster (256 GB unified) gemma4-26b-moe @ native on vLLM cluster TP=2 (triton, ROCm) batch ~411 @ 4K (200-conc. aggregate) — kyuz0 amd-strix-halo-vllm-toolboxes (triton cluster tp2 throughput, 200 reqs) ↗ I run this Be the first to run this 8× RTX Pro 6000 Blackwell server (768 GB) qwen3-5-397b-a17b-moe @ NVFP4 on SGLang+MTP ~350 @ 4K — local-inference-lab/rtx6kpro wiki (Qwen3.5-397B 8x single-batch) ↗ I run this Be the first to run this 8× H100 80 GB server qwen3-6-35b-a3b-moe @ FP8 on vLLM ~320 @ 128K — vLLM Recipes ↗ I run this Be the first to run this 2× Strix Halo cluster (256 GB unified) qwen3-6-35b-a3b-moe @ AWQ-4bit / native on vLLM cluster TP=2 (aiter, ROCm) batch ~287 @ 4K (200-conc. aggregate) — kyuz0 amd-strix-halo-vllm-toolboxes (aiter cluster tp2 throughput, Dec 2025) ↗ I run this Be the first to run this 2× Strix Halo cluster (256 GB unified) gpt-oss-120b @ MXFP4 on vLLM cluster TP=2 (triton, ROCm) batch ~229 @ 4K (200-conc. aggregate) — kyuz0 amd-strix-halo-vllm-toolboxes (triton cluster tp2 throughput, Dec 2025) ↗ I run this Be the first to run this Single RTX 5090 build qwen3-coder-30b @ Q4_K on llama.cpp (CUDA) ~226 @ 4K
~7093 @ 4K hardware-corner.net RTX 5090 LLM benchmarks (GGUF Q4) ↗ I run this Be the first to run this 8× H100 80 GB server mistral-medium-3-5-128b @ FP8 on vLLM ~220 @ 128K — vLLM Recipes ↗ I run this Be the first to run this Single RTX 5090 build gemma4-26b-moe @ Q4_K on llama.cpp (CUDA) ~180 @ 4K
~8799 @ 4K hardware-corner.net RTX 5090 LLM benchmarks (GGUF Q4) ↗ I run this Be the first to run this Single H100 80 GB workstation qwen3-6-35b-a3b-moe @ FP8 on vLLM ~180 @ 128K — vLLM Recipes ↗ I run this Be the first to run this 4× DGX Spark cluster (512 GB unified, CUDA) deepseek-v4-flash-284b-moe @ FP8 (FP8 KV, MTP n=2) on vLLM batch ~179.9 @ 393.216K (8-conc. aggregate) — NVIDIA Developer Forum 373808 (jasl vLLM TP=4, n=8 aggregate) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) qwen3-6-35b-a3b-moe @ AWQ-4bit / native on vLLM (aiter, ROCm) batch ~178 @ 4K (200-conc. aggregate) — kyuz0 amd-strix-halo-vllm-toolboxes (aiter tp1 throughput, Dec 2025) ↗ I run this Be the first to run this Single RTX Pro 6000 Blackwell 96 GB build qwen3-6-35b-a3b-moe @ FP8 on vLLM ~170 @ 262K — GitHub lastloop-ai ↗ I run this Be the first to run this Dual RTX Pro 6000 Blackwell build qwen3-6-27b-dense @ NVFP4 on vLLM+MTP ~156 @ 262K
~831 @ 262K loFT LLC ↗ I run this Be the first to run this Single RTX 3090 (used) build qwen3-coder-30b @ Q4_K on llama.cpp (CUDA) ~153 @ 4K
~2988 @ 4K hardware-corner.net RTX 3090 LLM benchmarks (GGUF Q4) ↗ I run this Be the first to run this Quad RTX Pro 6000 Blackwell build (384 GB) qwen3-5-397b-a17b-moe @ AWQ-INT4 (QuantTrio) on SGLang+MTP ~152 @ 4K — local-inference-lab/rtx6kpro wiki (Qwen3.5-397B single-batch decode) ↗ I run this Be the first to run this Dual RTX 3090 (used) build qwen3-6-35b-a3b-moe @ AWQ on vLLM ~149 @ 4K — GitHub - tfriedel (RTX 3090 lab) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) qwen3-6-35b-a3b-moe @ ROCmFP4 (CHADROCK) on llama-server+ROCmFPX ~140 @ 4K — GitHub hogeheer499-commits/strix-halo-guide ↗ I run this Be the first to run this Dual RTX Pro 6000 Blackwell build qwen3-6-27b-dense @ FP8 (native, fp8 KV, MTP spec=3) on vLLM ~137 @ 175K — theogravity/dual-rtx-6000-blackwell-qwen3.6-27b-fp8 (benchmark sweep) ↗ I run this Be the first to run this NVIDIA DGX Spark (128 GB) mistral-small-4-119b-moe @ NVFP4 on vLLM batch ~131 @ 262.144K (20-conc. aggregate) — Sebastien67 Medium (DGX Spark vLLM NVFP4, n=20 aggregate) ↗ I run this Be the first to run this Quad RTX Pro 6000 Blackwell build (384 GB) qwen3-5-397b-a17b-moe @ NVFP4 (nvidia checkpoint) on vLLM+MTP ~130 @ 4K — local-inference-lab/rtx6kpro wiki (Qwen3.5-397B MTP scaling table, concurrency=1) ↗ I run this Be the first to run this 2× Strix Halo cluster (256 GB unified) gemma-4-31b @ native (bf16/fp16) on vLLM cluster TP=2 (triton, ROCm) batch ~128 @ 4K (200-conc. aggregate) — kyuz0 amd-strix-halo-vllm-toolboxes (triton cluster tp2 throughput, 200 reqs) ↗ I run this Be the first to run this 4× DGX Spark cluster (512 GB unified, CUDA) minimax-m2-7-230b-moe @ MiniMax-M2.5-NVFP4 (modelopt_fp4, fp8 KV) on SGLang+MTP batch ~124 @ 196.608K (8-conc. aggregate) — NVIDIA Developer Forum 373676 (SGLang TP=4 EP=4, n=8 aggregate) ↗ I run this Be the first to run this Single RTX Pro 6000 Blackwell 96 GB build qwen3-next-80b-moe @ Q4_K_M on ollama / llama.cpp (CUDA) ~124 @ 4K
~3274 @ 4K vaditaslim.com RTX PRO 6000 Blackwell 8-model benchmarks ↗ I run this Be the first to run this Single RTX 3090 (used) build gemma4-26b-moe @ Q4_K on llama.cpp (CUDA) ~119 @ 4K
~3625 @ 4K hardware-corner.net RTX 3090 LLM benchmarks (GGUF Q4) ↗ I run this Be the first to run this Single H100 80 GB workstation qwen3-6-27b-dense @ FP16 on vLLM ~110 @ 128K — vLLM Recipes ↗ I run this Be the first to run this 2× Strix Halo cluster (256 GB unified) qwen3-5-122b-a10b-moe @ cyankiwi AWQ-4bit on vLLM cluster TP=2 (aiter, ROCm) batch ~104 @ 4K (200-conc. aggregate) — kyuz0 amd-strix-halo-vllm-toolboxes (aiter cluster tp2 throughput, Dec 2025) ↗ I run this Be the first to run this Single RTX 3090 (used) build qwen3-6-35b-a3b-moe @ Q4_K_XL on llama.cpp ~101 @ 65K
~1171 @ 0.5K aminrj.com (Qwen3.6 on 24GB) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) qwen3-6-35b-a3b-moe @ IQ4_XS-Q8nextn on llama-server+MTP ~101 @ 4K — GitHub hogeheer499-commits/strix-halo-guide ↗ I run this Be the first to run this 8× RTX Pro 6000 Blackwell server (768 GB) kimi-k2-5-1t-moe @ INT4 (BF16 KV, EP=8, overclocked GDDR7) on SGLang+MTP ~101 @ 4K — local-inference-lab/rtx6kpro wiki (Kimi K2.5 8x single-batch decode) ↗ I run this Be the first to run this Single RTX Pro 6000 Blackwell 96 GB build qwen3-6-27b-dense @ INT4 (AutoRound) + MTP n=3, FP8 KV on vLLM (flashinfer, MTP) ~100 @ 262K — GitHub lastloop-ai ↗ I run this Be the first to run this 8× RTX Pro 6000 Blackwell server (768 GB) glm-51-754b-moe @ NVFP4-MTP (lukealonso/GLM-5.1-NVFP4-MTP, served as GLM-5) on SGLang+MTP ~100 @ 4K — local-inference-lab/rtx6kpro wiki (GLM-5 single-batch decode; models/glm5.md = GLM-5.1) ↗ I run this Be the first to run this NVIDIA DGX Spark (128 GB) qwen3-6-35b-a3b-moe @ NVFP4 on vLLM+DFlash ~97 @ 0.5K
~9090 @ 0.5K (derived) GitHub AEON-7 ↗ I run this Be the first to run this MacBook Pro M5 Max 64 GB gemma-4-12b @ MLX NVFP4 on Ollama 0.31 (MLX) + MTP ~95 @ 4K — Ollama blog (framework-author first-party; M5 Max, Aider polyglot) ↗ I run this Be the first to run this Single RTX 5090 build qwen3-6-27b-dense @ NVFP4 on vLLM ~92 @ 200K
~5300 @ 47K GitHub devnen ↗ I run this Be the first to run this NVIDIA DGX Spark (128 GB) qwen3-6-35b-a3b-moe @ NVFP4 on vLLM ~90 @ 43K
~2133 @ 32K (derived) GitHub technigmaai/dgx-spark ↗ I run this Be the first to run this Dual RTX 3090 (used) build qwen3-6-27b-dense @ AWQ on vLLM ~90 @ 100K — GitHub - tfriedel (RTX 3090 lab) ↗ I run this Be the first to run this MacBook Pro M5 Max 64 GB qwen3-6-35b-a3b-moe @ MLX-4bit on MLX-LM ~87 @ 4K
~2447 @ 4K oMLX Benchmark ↗ I run this Be the first to run this Dual RTX Pro 6000 Blackwell build minimax-m2-7-230b-moe @ MiniMax-M2.5-NVFP4 on vLLM ~85 @ 4K — local-inference-lab/rtx6kpro wiki (MiniMax-M2.5 single-stream table) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) qwen3-6-35b-a3b-moe @ Q4_0 on llama.cpp ~81 @ 4K
~1244 @ 0.5K GitHub hogeheer499-commits/strix-halo-guide ↗ I run this Be the first to run this Quad RTX Pro 6000 Blackwell build (384 GB) minimax-m2-7-230b-moe @ MiniMax-M2.5-FP8 on vLLM ~81 @ 20K — local-inference-lab/rtx6kpro wiki (MiniMax-M2.5 single-stream table) ↗ I run this Be the first to run this DGX B200 — 8× B200 server (1.44 TB HBM3e) nemotron-3-ultra-550b-a55b-moe @ NVFP4 + FP8 KV on Dynamo + vLLM (TP=4, expert-parallel, MTP) batch ~80.6 @ 4K (20-conc. aggregate) — NVIDIA ai-dynamo/dynamo recipes (B200 TP4+EP, NVFP4+FP8, MTP) ↗ I run this Be the first to run this 4× DGX Spark cluster (512 GB unified, CUDA) minimax-m3-428b-moe @ MiniMax-M3-AWQ-INT4 (fp8 KV, EAGLE3) on vLLM batch ~79 @ 262.144K (4-conc. aggregate) — NVIDIA Developer Forum 375361 (vLLM TP=4, n=4 aggregate) ↗ I run this Be the first to run this Single AMD Radeon AI Pro R9700 32 GB build qwen3-6-35b-a3b-moe @ Q4_K_M on llama.cpp ~77 @ 4K
~1636 @ 32K GitHub truelies444 ↗ I run this Be the first to run this 8× H100 80 GB server kimi-k2-6-1t-moe @ FP8 on vLLM ~75 @ 256K — HF - RedHatAI (Kimi-K2.6-FP8-BLOCK) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) qwen3-6-35b-a3b-moe @ MTP-GGUF UD-Q4_K_XL (draft-mtp n=3) on llama.cpp (Vulkan RADV, MTP) ~75 @ 0.5K — kyuz0 amd-strix-halo-toolboxes MTP grid (results-mtp/summary.json, 15 May 2026) ↗ I run this Be the first to run this 2× Strix Halo cluster (256 GB unified) qwen3-5-122b-a10b-moe @ cyankiwi AWQ-8bit on vLLM cluster TP=2 (aiter, ROCm) batch ~74 @ 4K (200-conc. aggregate) — kyuz0 amd-strix-halo-vllm-toolboxes (aiter cluster tp2 throughput, 200 reqs) ↗ I run this Be the first to run this Dual AMD Radeon AI Pro R9700 build (64 GB) qwen3-6-35b-a3b-moe @ Q6_K on llama.cpp ~72 @ 4K
~3038 @ 32K GitHub truelies444 ↗ I run this Be the first to run this Single RTX 3090 (used) build qwen3-6-27b-dense @ AWQ/AutoRound-INT4 on vLLM+MTP ~72 @ 32K — GitHub devnen ↗ I run this Be the first to run this Single RTX 4090 build qwen3-coder-30b @ Q4_K on llama.cpp (CUDA) ~68 @ 64K
~1502 @ 64K hardware-corner.net RTX 4090 LLM benchmarks (GGUF Q4) ↗ I run this Be the first to run this 2× DGX Spark cluster (256 GB unified, CUDA) deepseek-v4-flash-284b-moe @ not stated (native precision) on vLLM (Aidendle94/B12X-MoE, TP=2 RoCE) + DSpark spec-decode ~65 @ 200K — GitHub 0rand (DeepSeek-V4 DSpark serving stack) ↗ I run this Be the first to run this 2× DGX Spark cluster (256 GB unified, CUDA) deepseek-v4-flash-284b-moe @ NVFP4-KV (nvfp4_ds_mla) on vLLM+DSpark ~63 @ 200K — NVIDIA Developer Forum (374846) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) qwen3-6-35b-a3b-moe @ Q4_K_M (UD) on llama.cpp ~62 @ 4K
~1059 @ 0.5K GitHub hogeheer499-commits/strix-halo-guide ↗ I run this Be the first to run this 8× Strix Halo cluster (1024 GB unified) qwen3-6-35b-a3b-moe @ Q8_0 on llama.cpp ~62 @ 4K — GitHub - strix-halo-guide ↗ I run this Be the first to run this NVIDIA DGX Spark (128 GB) qwen3-6-35b-a3b-moe @ FP8 on vLLM ~60 @ 32K
~6520 @ 8K (derived) NVIDIA Developer Forum (366822) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) qwen3-6-35b-a3b-moe @ UD-Q4_K_XL on llama.cpp (Vulkan RADV) ~60 @ 0.5K
~1114 @ 0.5K kyuz0 amd-strix-halo-toolboxes grid (docs/results.json, 16 May 2026) ↗ I run this Be the first to run this DGX H200 — 8× H200 server (1.13 TB HBM3e) nemotron-3-ultra-550b-a55b-moe @ NVFP4 + FP8 KV on Dynamo + vLLM (TP=8, expert-parallel, MTP) batch ~58.7 @ 4K (10-conc. aggregate) — NVIDIA ai-dynamo/dynamo recipes (8xH200 TP8+EP, NVFP4+FP8, MTP) ↗ I run this Be the first to run this 2× Strix Halo cluster (256 GB unified) minimax-m2-7-230b-moe @ cyankiwi AWQ-4bit on vLLM cluster TP=2 (aiter, ROCm) batch ~57 @ 4K (200-conc. aggregate) — kyuz0 amd-strix-halo-vllm-toolboxes (aiter cluster tp2 throughput, 200 reqs) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) gpt-oss-120b @ MXFP4 on llama.cpp (Vulkan RADV) ~56 @ 0.5K
~720 @ 0.5K kyuz0 amd-strix-halo-toolboxes grid (docs/results.json, 16 May 2026) ↗ I run this Be the first to run this Single Intel Arc Pro B70 build qwen3-6-35b-a3b-moe @ Q4_K_M (UD) on llama.cpp (SYCL) ~55 @ 4K
~615 @ 0.5K GitHub PMZFX ↗ I run this Be the first to run this Tesla V100 32 GB SXM2 mod build qwen3-6-35b-a3b-moe @ Q5_K_M on llama.cpp ~55 @ 10K — GitHub - ai-bond (V100 flash-attn) ↗ I run this Be the first to run this 2× DGX Spark cluster (256 GB unified, CUDA) mimo-v2-5-310b-a15b-moe @ NVFP4 on vLLM+DFlash ~54 @ 0.5K
~2083 @ 50K (derived) GitHub HeNryous (renek) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) gemma4-26b-moe @ UD-Q4_K_XL on llama.cpp (Vulkan RADV) ~54 @ 0.5K
~1324 @ 0.5K kyuz0 amd-strix-halo-toolboxes grid (docs/results.json, 16 May 2026) ↗ I run this Be the first to run this MacBook Pro M5 Max 64 GB gemma-4-12b @ MLX NVFP4 on Ollama 0.31 (MLX), no MTP ~50.2 @ 4K — Ollama blog (framework-author first-party; M5 Max) ↗ I run this Be the first to run this NVIDIA DGX Spark (128 GB) qwen3-6-35b-a3b-moe @ FP8 on vLLM+DFlash ~50 @ 262K
~4932 @ 0.5K GitHub ZengboJamesWang ↗ I run this Be the first to run this 4× Strix Halo cluster (512 GB unified) qwen3-6-35b-a3b-moe @ Q8_0 on llama.cpp ~50 @ 4K — Frame.work Community ↗ I run this Be the first to run this 4× DGX Spark cluster (512 GB unified, CUDA) deepseek-v4-flash-284b-moe @ FP8 (official weights, FP8 KV, MTP n=2) on vLLM ~49.4 @ 393.216K — NVIDIA Developer Forum 373808 (jasl vLLM TP=4) ↗ I run this Be the first to run this 2× DGX Spark cluster (256 GB unified, CUDA) deepseek-v4-flash-284b-moe @ FP8 on vLLM+MTP ~45.5 @ 1000K
~786 @ 800K GitHub tonyd2wild ↗ I run this Be the first to run this 2× DGX Spark cluster (256 GB unified, CUDA) mimo-v2-5-310b-a15b-moe @ NVFP4 on vLLM+DFlash ~45 @ 131K — NVIDIA Developer Forum (375607) ↗ I run this Be the first to run this 2× DGX Spark cluster (256 GB unified, CUDA) mimo-v2-5-310b-a15b-moe @ NVFP4 on vLLM+DFlash ~45 @ 26K
~5540 @ 0.256K (derived) NVIDIA Developer Forum (375923) ↗ I run this Be the first to run this Single H100 80 GB workstation mistral-medium-3-5-128b @ FP8 on vLLM ~45 @ 128K — vLLM Recipes ↗ I run this Be the first to run this 2× DGX Spark cluster (256 GB unified, CUDA) deepseek-v4-flash-284b-moe @ FP8 on vLLM+MTP ~40 @ 500K — GitHub tonyd2wild ↗ I run this Be the first to run this 8× DGX Spark cluster (1024 GB unified, CUDA) mimo-v2-5-pro-1t-moe @ NVFP4 on vLLM+MTP ~40 @ 1K
~1950 @ 2K NVIDIA Developer Forum (370803) ↗ I run this Be the first to run this 2× Strix Halo cluster (256 GB unified) qwen3-6-35b-a3b-moe @ Q8_0 on llama.cpp ~40 @ 4K — Frame.work Community ↗ I run this Be the first to run this 8× DGX Spark cluster (1024 GB unified, CUDA) qwen3-5-397b-a17b-moe @ FP8 (406 GiB) on vLLM ~39.5 @ 32K — NVIDIA Developer Forum 369446 (vLLM eugr fork TP=8) ↗ I run this Be the first to run this NVIDIA DGX Spark (128 GB) qwen3-6-27b-dense @ Q4_K_M on llama.cpp+DFlash ~38 @ 256K — GitHub phuongncn ↗ I run this Be the first to run this 2× DGX Spark cluster (256 GB unified, CUDA) mimo-v2-5-310b-a15b-moe @ NVFP4 4-bit weights + NVFP4 4-bit KV cache on vLLM (TP=2 Ray) + DFlash spec-decode ~37.8 @ 1000K — GitHub tonyd2wild (MiMo V2.5 DFlash 1M NVFP4-KV) ↗ I run this Be the first to run this 4× DGX Spark cluster (512 GB unified, CUDA) qwen3-5-397b-a17b-moe @ NVFP4 on vLLM ~37 @ 32K — Level1Techs Forum ↗ I run this Be the first to run this 4× DGX Spark cluster (512 GB unified, CUDA) minimax-m3-428b-moe @ MiniMax-M3-MXFP4 (bf16 KV, EAGLE3 k=2) on vLLM ~34.8 @ 262.144K
~2020 @ 262.144K NVIDIA Developer Forum 375386 (vLLM TP=4 EAGLE3) ↗ I run this Be the first to run this 2× DGX Spark cluster (256 GB unified, CUDA) mimo-v2-5-310b-a15b-moe @ NVFP4 on vLLM+MTP ~34 @ 0.5K
~2609 @ 2K (derived) NVIDIA Developer Forum (370459) ↗ I run this Be the first to run this 4× DGX Spark cluster (512 GB unified, CUDA) minimax-m3-428b-moe @ MiniMax-M3-AWQ-INT4 (fp8 KV, EAGLE3) on vLLM ~33.7 @ 262.144K — NVIDIA Developer Forum 375361 (vLLM TP=4 EAGLE3) ↗ I run this Be the first to run this 2× DGX Spark cluster (256 GB unified, CUDA) deepseek-v4-flash-284b-moe @ FP8 on vLLM ~33 @ 32K
~512 @ 128K (derived) NVIDIA Developer Forum (370309) ↗ I run this Be the first to run this 8× H100 80 GB server deepseek-v3-671b-moe @ FP8 (671B, TP=8) on vLLM ~33 @ 1.024K — dzhsurf/deepseek-v3-r1-deploy-and-benchmarks (8xH100 vLLM TP=8, concurrency=1) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) minimax-m2-7-230b-moe @ UD-Q3_K_S on llama.cpp (Vulkan RADV) ~31 @ 0.5K
~243 @ 0.5K kyuz0 amd-strix-halo-toolboxes grid (docs/results.json, 16 May 2026) ↗ I run this Be the first to run this Dual RTX Pro 6000 Blackwell build mistral-medium-3-5-128b @ FP8 on vLLM ~30 @ 98K — HF mistralai discussion #17 ↗ I run this Be the first to run this Quad Tesla P40 (96 GB) homelab build gpt-oss-120b @ MXFP4 (native MoE, ~65GB, fits in 96GB across 4x P40) on llama.cpp ~28.1 @ 4K — TinyComputers.io (Tesla P40 home lab, gpt-oss-120b MXFP4) ↗ I run this Be the first to run this NVIDIA DGX Spark (128 GB) qwen3-6-27b-dense @ Q4_K_M on llama.cpp+MTP ~28 @ 2K
~1084 @ 2K NVIDIA Developer Forum (370298) ↗ I run this Be the first to run this NVIDIA DGX Spark (128 GB) mistral-small-4-119b-moe @ NVFP4 on vLLM ~27.8 @ 262.144K — Sebastien67 Medium (first-hand DGX Spark vLLM NVFP4 run) ↗ I run this Be the first to run this 2× Strix Halo cluster (256 GB unified) minimax-m2-7-230b-moe @ MiniMax-M2.5-REAP Q4_K_M (GGUF, pruned REAP variant) on llama.cpp (Vulkan RADV, RPC cluster) ~26.7 @ 0.512K
~272 @ 0.512K visorcraft/strix-halo-llm-perf (2-node RPC llama-bench, 2026-02-19) ↗ I run this Be the first to run this 4× DGX Spark cluster (512 GB unified, CUDA) minimax-m2-7-230b-moe @ MiniMax-M2.5-NVFP4 (modelopt_fp4, fp8 KV) on SGLang+MTP ~25.5 @ 196.608K — NVIDIA Developer Forum 373676 (SGLang TP=4 EP=4) ↗ I run this Be the first to run this Mac Mini M4 (24 GB) qwen3-6-35b-a3b-moe @ MLX-4bit on MLX-LM ~25 est. @ 4K — maloyan.xyz (M4 16GB, scaled) ↗ I run this Be the first to run this 2× Strix Halo cluster (256 GB unified) minimax-m2-7-230b-moe @ MiniMax-M2.5-REAP MXFP4_MOE (GGUF) on llama.cpp (Vulkan RADV, RPC cluster) ~24.5 @ 0.512K
~299.5 @ 0.512K visorcraft/strix-halo-llm-perf (2-node RPC llama-bench, 2026-02-19) ↗ I run this Be the first to run this Single AMD Radeon AI Pro R9700 32 GB build qwen3-6-27b-dense @ Q5_K_M on llama.cpp ~24 @ 4K
~611 @ 32K GitHub truelies444 ↗ I run this Be the first to run this Dual AMD Radeon AI Pro R9700 build (64 GB) qwen3-6-35b-a3b-moe @ base weights (fp8 KV) on vLLM ~22.8 @ 1.036K
~4600 @ 1.036K mlai.blog (Qwen3.5-35B-A3B on dual R9700, ROCm vLLM) ↗ I run this Be the first to run this 4× DGX Spark cluster (512 GB unified, CUDA) glm-5-2-753b-moe @ AWQ-INT4 (cyankiwi/GLM-5.2-AWQ-INT4, 15% data-free expert pruning) on vLLM+MTP ~22 @ 8.192K
~535 @ 8.192K NVIDIA Developer Forum 374125 (CosmicRaisins, AWQ-INT4 TP=4 MTP) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) qwen3-5-122b-a10b-moe @ UD-Q5_K_XL on llama.cpp (Vulkan RADV) ~22 @ 0.5K
~337 @ 0.5K kyuz0 amd-strix-halo-toolboxes grid (docs/results.json, 16 May 2026) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) qwen3-6-27b-dense @ UD-Q4_K_M (draft-mtp n=3) on llama.cpp (ROCm, MTP) ~21 @ 0.5K — Caleb Coffie - benchmarking llama.cpp MTP on Strix Halo ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) llama-4-scout @ UD-Q4_K_XL on llama.cpp (Vulkan RADV) ~20 @ 0.5K
~103 @ 0.5K hardware-corner.net Strix Halo optimization benchmarks ↗ I run this Be the first to run this 8× DGX Spark cluster (1024 GB unified, CUDA) kimi-k2-6-1t-moe @ NVFP4 on vLLM ~18 @ 32K — NVIDIA Developer Forum ↗ I run this Be the first to run this Mac Mini M4 (16 GB) qwen3-6-35b-a3b-moe @ MLX-4bit on MLX-LM ~17 @ 4K — maloyan.xyz (M4 16GB) ↗ I run this Be the first to run this Tesla V100 32 GB SXM2 mod build qwen3-6-27b-dense @ Q5_K_M on llama.cpp ~17 @ 32K — hardware-corner.net (V100 32GB guide) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) qwen3-6-27b-dense @ Q5_K_M on llama.cpp ~16 @ 4K — llm-tracker.info (kyuz0) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) deepseek-v4-flash-284b-moe @ IQ2_XXS-w2Q2K imatrix (~80.8 GB) on ds4 (antirez DeepSeek-V4-Flash engine) + MTP, ROCm 7.2.4 gfx1151 ~15.25 @ 2K
~152 @ 2K (derived) kyuz0 ds4 Strix Halo toolbox (ds4-bench, single 128GB node); antirez ds4 engine ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) deepseek-v4-flash-284b-moe @ Hybrid Q2/Q4 imatrix (layers 37-42 Q4, ~97 GB) on ds4 (antirez engine) + MTP, ROCm 7.2.4 gfx1151 ~15.02 @ 2K
~138 @ 2K kyuz0 ds4 Strix Halo toolbox (ds4-bench, single-node hybrid Q2/Q4) ↗ I run this Be the first to run this MacBook Air M4 (16 GB) qwen3-6-35b-a3b-moe @ MLX-4bit on MLX-LM ~15 est. @ 4K — maloyan.xyz (M4 16GB, fanless-adjusted) ↗ I run this Be the first to run this 4× DGX Spark cluster (512 GB unified, CUDA) glm-5-2-753b-moe @ NVFP4 (REAP-less, high-quality 4-bit; cf. nvidia/GLM-5.2-NVFP4 model card) on vLLM ~15 @ 131.072K
~500 @ 131.072K NVIDIA Developer Forum 374832 (REAP-less NVFP4, custom vLLM fork TP=4) ↗ I run this Be the first to run this Quad RTX 3090 (used) build devstral-2-123b @ IQ4_KSS (GGUF) on ik_llama.cpp (-sm graph, tensor-parallel) ~15 est. @ 4K
~300 @ 2K HF ubergarm Devstral-2-123B-GGUF discussion #2 (phakio, ik_llama.cpp 4-GPU) ↗ I run this Be the first to run this 2× Strix Halo cluster (256 GB unified) mimo-v2-5-310b-a15b-moe @ UD-Q4/Q5_K_XL GGUF (~180-215 GB, split across 2 nodes) on llama.cpp RPC (2x Strix Halo 128GB, ROCm, USB4net secondary link) ~15 @ 10K
~356 @ 10K (derived) r/LocalLLaMA operator report (2x Strix Halo 128GB, llama.cpp RPC over USB4net); AesSedai/unsloth MiMo-V2.5 GGUF ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) nemotron-3-super-120b-a12b-moe @ UD-Q4_K_XL on llama.cpp (ROCm 7.2.3) ~14 @ 0.5K
~276 @ 0.5K kyuz0 amd-strix-halo-toolboxes grid (docs/results.json, 16 May 2026) ↗ I run this Be the first to run this 8× DGX Spark cluster (1024 GB unified, CUDA) kimi-k2-6-1t-moe @ NVFP4 (60 shards, ~554 GiB, no spec-decode) on vLLM ~13.5 @ 32.768K — NVIDIA Developer Forum 369446 (vLLM eugr fork TP=8) ↗ I run this Be the first to run this 2× Strix Halo cluster (256 GB unified) deepseek-v4-flash-284b-moe @ Q4 imatrix distributed (~153.3 GB, Q4 experts) on ds4 multi-node (pipeline-parallel, 2x Strix, ROCm 7.2.4 gfx1151) + MTP ~13.01 @ 2K
~62 @ 2K (derived) kyuz0 ds4 Strix Halo toolbox (ds4-bench, 2-node distributed Q4) ↗ I run this Be the first to run this Dual RTX 3090 (used) build gpt-oss-120b @ Q4_K_M (--n-cpu-moe 26, tensor-split 0.5/0.5) on llama.cpp (CUDA, 2x GPU + CPU-MoE) ~13 @ 4K — LLM Garage - GPT-OSS-120B on Dual RTX 3090s ↗ I run this Be the first to run this 2× DGX Spark cluster (256 GB unified, CUDA) glm-5-2-753b-moe @ low-bit (vLLM TP=2) on vLLM (TP over 2 Sparks) ~12 @ 40K — NVIDIA Developer Forum 374523 (GLM-5.2 vLLM TP=2 update) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) gemma-4-31b @ UD-Q4_K_XL on llama.cpp (Vulkan RADV) ~11 @ 0.5K
~302 @ 0.5K kyuz0 amd-strix-halo-toolboxes grid (docs/results.json, 16 May 2026) ↗ I run this Be the first to run this 2× DGX Spark cluster (256 GB unified, CUDA) glm-5-2-753b-moe @ UD-IQ1_S (~2.3 bpw, 1-bit) on llama.cpp RPC (tensor-split over 2 nodes) ~8 @ 2K
~213 @ 2K NVIDIA Developer Forum 374523 (GLM-5.2 on 2x DGX Spark, 1-bit llama.cpp RPC) ↗ I run this Be the first to run this 4× DGX Spark cluster (512 GB unified, CUDA) glm-5-2-753b-moe @ IQ4_XS (GGUF, ~365GB across 4 nodes, DSA sparse attention active) on llama.cpp (RPC multi-node) ~6.28 @ 1048.576K
~222 @ 1048.576K NVIDIA Developer Forum 373933 (IQ4_XS llama.cpp RPC, DSA active) ↗ I run this Be the first to run this RTX 3060 12 GB build gemma-4-12b @ Q4_K_M (GGUF) on llama.cpp (Vulkan) ~5 @ 4K — Hacker News Gemma 4 12B launch thread (community, Vulkan-throttled) ↗ I run this Be the first to run this 8× Strix Halo cluster (1024 GB unified) kimi-k2-6-1t-moe @ Q5_K_M on llama.cpp ~5 @ 4K — Frame.work Community ↗ I run this Be the first to run this Quad Tesla P40 (96 GB) homelab build mistral-medium-3-5-128b @ Q4_K_M on llama.cpp ~4 @ 32K — GitHub - llama.cpp #12990 (P40 FA) ↗ I run this Be the first to run this 2× Strix Halo cluster (256 GB unified) mistral-medium-3-5-128b @ Q4_K_M on llama.cpp ~3 @ 4K — llm-tracker.info (kyuz0) ↗ I run this Be the first to run this Single RTX 3090 (used) build gpt-oss-120b @ MXFP4 (F16 GGUF, --n-cpu-moe offload) on llama.cpp (CUDA + CPU-MoE offload) ~2 @ 0.1K
~48 @ 0.1K hardware-corner.net gpt-oss CPU-MoE offloading benchmark ↗ I run this Be the first to run this Single AMD Instinct MI50 32 GB (used) build qwen3-6-35b-a3b-moe @ Q4_K_M on llama.cpp — — GitHub llama.cpp #19880 (MI50 enablement) ↗ I run this Be the first to run this Quad AMD MI50 32 GB (128 GB) homelab build qwen3-6-35b-a3b-moe @ Q5_K_M on llama.cpp — — aibytes.blog (MI50 ROCm vs Vulkan) ↗ I run this Be the first to run this Quad AMD MI50 32 GB (128 GB) homelab build mistral-medium-3-5-128b @ Q4_K_M on llama.cpp — — HF - unsloth (GGUF discussion) ↗ I run this Be the first to run this Mac Mini M4 (16 GB) gemma-4-12b @ MLX 4-bit on Ollama (MLX) / mlx-lm — — Ollama model library (Apple-Silicon MLX build; runnable recipe). Fit corroborated by Gemma 4 launch HN thread (Q4_K_M ~6.6 GB). ↗ I run this Be the first to run this Mac Mini M4 (24 GB) gemma-4-12b @ MLX 4-bit on Ollama (MLX) / mlx-lm — — Ollama model library (Apple-Silicon MLX build). Fit corroborated by HN launch thread. ↗ I run this Be the first to run this MacBook Air M4 (16 GB) gemma-4-12b @ MLX 4-bit on Ollama (MLX) / mlx-lm — — Ollama model library (Apple-Silicon MLX build). Fit corroborated by HN launch thread. ↗ I run this Be the first to run this NVIDIA DGX Spark (128 GB) qwen3-6-35b-a3b-moe @ NVFP4 on SGLang+MTP — — GitHub r0b0tlab ↗ I run this Be the first to run this 2× DGX Spark cluster (256 GB unified, CUDA) deepseek-v4-flash-284b-moe @ NVFP4-KV (nvfp4_ds_mla) on vLLM+DSpark — — HF drowzeys ↗ I run this Be the first to run this 2× DGX Spark cluster (256 GB unified, CUDA) qwen3-6-35b-a3b-moe @ FP8 on vLLM (Ray TP=2) — — Medium - Michael Peres ↗ I run this Be the first to run this 4× DGX Spark cluster (512 GB unified, CUDA) deepseek-v4-flash-284b-moe @ FP8 on vLLM — — NVIDIA Developer Forum ↗ I run this Be the first to run this 4× DGX Spark cluster (512 GB unified, CUDA) nemotron-3-ultra-550b-a55b-moe @ NVFP4 (FP8 KV, MTP) on vLLM (TP=4) — — NVIDIA-NeMo Nemotron Spark Deployment Guide (4x DGX Spark, NVFP4, vLLM TP=4) ↗ I run this Be the first to run this 8× DGX Spark cluster (1024 GB unified, CUDA) deepseek-v4-pro-1t6-moe @ FP8 on vLLM — — GitHub - vLLM #43367 ↗ I run this Be the first to run this Single Intel Arc B580 12 GB build qwen3-6-27b-dense @ Q4_K_M on llama.cpp — — GitHub - intel/llm-scaler (official) ↗ I run this Be the first to run this Quad RTX 3090 (used) build mistral-medium-3-5-128b @ Q4_K_M on llama.cpp — — HF - bartowski (GGUF) ↗ I run this Be the first to run this Quad RTX 3090 (used) build qwen3-6-35b-a3b-moe @ FP16 on vLLM — — GitHub - tfriedel (RTX 3090 lab) ↗ I run this Be the first to run this Single RTX 4090 build qwen3-6-27b-dense @ AWQ on vLLM — — GitHub - thc1006 (Ampere/Ada spec-decode) ↗ I run this Be the first to run this Single RTX 4090 build qwen3-6-35b-a3b-moe @ Q4_K_M on llama.cpp — — GitHub - thc1006 (spec-decode) ↗ I run this Be the first to run this RTX 3060 12 GB build qwen3-6-27b-dense @ Q4_K_M on llama.cpp — — GitHub - llama.cpp build docs ↗ I run this Be the first to run this Dual RTX 5090 build qwen3-6-35b-a3b-moe @ NVFP4 on vLLM — — HF - RedHatAI (NVFP4) ↗ I run this Be the first to run this Dual RTX 5090 build qwen3-6-27b-dense @ FP8 on vLLM — — HF - Qwen (official FP8) ↗ I run this Be the first to run this Single RTX Pro 6000 Blackwell 96 GB build mistral-medium-3-5-128b @ NVFP4 on vLLM — — HF nvidia (NVFP4 card) ↗ I run this Be the first to run this Single Tesla P40 24 GB (used) build qwen3-6-27b-dense @ Q4_K_M on llama.cpp — — GitHub llama.cpp #19248 (P40) ↗ I run this Be the first to run this Quad Tesla P40 (96 GB) homelab build qwen3-6-35b-a3b-moe @ Q5_K_M on llama.cpp — — GitHub - llama.cpp #12990 (P40 FA) ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) mimo-v2-5-310b-a15b-moe @ UD-IQ2_M (~2.7 bpw, ~92.8 GB) on llama.cpp (Vulkan/RADV, kyuz0 container), gfx1151 —
~31 @ 0.5K (derived) hogeheer499 strix-halo-guide community evidence map (Corsair AI WS 300, IQ2_M, capacity row); bartowski MiMo GGUF ↗ I run this Be the first to run this AMD Ryzen AI Max+ 395 (128 GB) gemma-4-12b @ UD-Q4_K_XL (GGUF) on llama.cpp (ROCm 7.2.x, gfx1151) — — kyuz0 Strix Halo toolboxes (ROCm gfx1151 llama.cpp recipe; 12B not yet in grid) ↗ I run this Be the first to run this 4× Strix Halo cluster (512 GB unified) qwen3-5-397b-a17b-moe @ Q-family GGUF (HIP+RPC, np2, ctx 200k) on llama.cpp — — visorcraft/strix-halo-llm-perf (Qwen3.5-397B RPC shape-control, 2026-02-21) ↗ I run this Be the first to run this 4× Strix Halo cluster (512 GB unified) deepseek-v4-flash-284b-moe @ Q4_K-class across 4 nodes on llama.cpp RPC (4x Framework Desktop / Strix mainboards) — — frame.work llama.cpp RPC multi-node recipe (extended to 4 nodes) ↗ I run this Be the first to run this 4× Strix Halo cluster (512 GB unified) mimo-v2-5-310b-a15b-moe @ Q4_K_M (~178 GB) / Q5_K_M (~213 GB) on llama.cpp RPC (4x Framework Desktop / Strix mainboards) — — bartowski MiMo-V2.5 GGUF (Q4_K_M/Q5) + frame.work llama.cpp RPC 4-node ↗ I run this Be the first to run this MacBook Pro M5 Pro 48 GB qwen3-6-35b-a3b-moe @ MLX-4bit on MLX-LM — — GitHub ml-explore/mlx-lm ↗ I run this Be the first to run this MacBook Pro M5 Pro 48 GB gemma-4-12b @ MLX 4-bit on Ollama 0.31 (MLX) + MTP — — Ollama blog (framework-author first-party MTP recipe; M5-family). Directional only; M5 Pro not separately measured. ↗ I run this Be the first to run this MacBook Pro M5 Pro 48 GB qwen3-6-27b-dense @ MLX 4-bit on mlx-lm / Ollama (MLX) — — Ollama model library (Apple-Silicon MLX build). Fit from db.json sizeQ4=16 vs 40 GB usable. ↗ I run this Be the first to run this DGX H200 — 8× H200 server (1.13 TB HBM3e) kimi-k2-7-code-1t-moe @ native INT4 on vLLM (TP=8, expert-parallel) — — vLLM Recipes (official K2.7 Code command, 8xH200 INT4); tok/s = K2.6 same-box SGLang INT4 (paxsaroffcuts) ↗ I run this Be the first to run this 
LLMRequirements.com