Best AI models that run on 12× RTX Pro 6000 Blackwell rack (1152 GB)

Datacenter-class memory pool on PCIe parts.

NVIDIA · 8U server rack (multi-node, 1-2 chassis)

12× RTX Pro 6000 Blackwell rack (1152 GB)

1152 GB 1116 GB usable 1792 GB/s $118k

Amazon CoreWeave SuperMicro SHI enterprise

What models fit this build

12× RTX Pro 6000 Blackwell rack (1152 GB) has 1116 GB of usable memory. Here's which open-weights model sizes fit at each quant, at ~8K context with an FP8 KV cache.

Largest comfortable fit: 1T+ MoE at Q8 (~1063 GB).

Model size	Q2	Q4	Q5	Q8
7–8B	Fits	Fits	Fits	Fits
13–14B	Fits	Fits	Fits	Fits
30–32B	Fits	Fits	Fits	Fits
70–72B	Fits	Fits	Fits	Fits
~120B MoE	Fits	Fits	Fits	Fits
~235B MoE	Fits	Fits	Fits	Fits
~670B MoE	Fits	Fits	Fits	Fits
1T+ MoE	Fits	Fits	Fits	Fits

✓ = the weights + FP8 KV cache fit within this build's usable memory at ~8K context. Longer context needs more — size any model in the picker →

Closed-frontier reference (May 2026)

What "API-grade" actually scores right now. Use this as the ceiling — anything local will lag here by some margin, and that's fine for most workflows.

Google DeepMind 2026-02

Gemini 3.1 Pro

2.0M ctx $2 / $12 per M tok

API: 125 t/s out · +30 s thinking

HLE44.7%
Terminal-Bench 280.2%
SWE-Bench Pro54.2%
SWE-Bench Ver80.6%
Aider Polyglot—
LiveCodeBench91.7%
GPQA Diamond94.3%
MMLU-Pro91.0%

Coding: Coding-strong, especially repo-level work with 2M context. Less of a Cursor/Cline default than Claude — Gemini Code Assist users prefer it. Top of leaderboards on GPQA + LCB. Now #7 on OpenRouter coding by volume (120B tokens) — Gemini 3.5 Flash will displace it next month.

Agent: TB2 80.2 makes it agent-grade. Used widely on Google's Agent Builder; less common in Open-Claude/Hermes. Reliable in long Vertex-AI Agent runs (multi-hour).

OpenAI 2026-04

ChatGPT 5.5

922K ctx $5 / $30 per M tok

API: 61 t/s out · +28 s thinking

HLE52.2%
Terminal-Bench 282.0%
SWE-Bench Pro58.6%
SWE-Bench Ver88.7%
Aider Polyglot88.0%
LiveCodeBench—
GPQA Diamond93.6%
MMLU-Pro—

Coding: Codex CLI + GPT-5.5 is the top of Terminal-Bench. r/ChatGPTCoding has shifted to it for daily coding; Cursor users mixed (some prefer Claude Sonnet 4.6 for diff quality).

Agent: Strongest published agent score (TB2 82.0%, re-verified 2026-05-14). Widely used in OpenAI Assistants, AutoGPT-style swarms, and Open-Claude routing. Reliable in 4-8h autonomous sessions.

Anthropic 2026-06

Claude Sonnet 5

1.0M ctx $3 / $15 per M tok

HLE57.4%
Terminal-Bench 280.4%
SWE-Bench Pro63.2%
SWE-Bench Ver—
Aider Polyglot—
LiveCodeBench—
GPQA Diamond—
MMLU-Pro—

Coding: Anthropic's 2026-06-30 release: near-Opus intelligence at the old Sonnet price. Terminal-Bench 2.1 jumps to 80.4 (from Sonnet 4.6's 53.4), putting it in the top tier of agentic coders next to GPT-5.5 (82) and Gemini 3.1 Pro (80.2). 1M context, adaptive thinking on by default; the daily workhorse for Cursor / Zed / Cline users who want closed-model diff quality.

Agent: TB2 80.4 makes Sonnet 5 a top-tier agent driver (Sonnet 4.6 was mid-tier at 53.4). SWE-Bench Pro 63.2 lands between GPT-5.5 (58.6) and Opus 4.7 (69.2). Anthropic's tool-use SDK keeps it the most reliable closed model for hand-rolled agent loops.

Anthropic 2026-05

Claude Opus 4.8

1.0M ctx $5 / $25 per M tok

API: 59 t/s out · +22 s thinking

HLE57.9%
Terminal-Bench 2—
SWE-Bench Pro69.2%
SWE-Bench Ver88.6%
Aider Polyglot—
LiveCodeBench—
GPQA Diamond93.6%
MMLU-Pro—

Coding: Cursor / Cline / Zed power-user pick when budget allows. SWE-Pro 69.2 and TB2.1 74.6 say it all — best closed model for real software engineering. Simon Willison describes it as 'a modest but tangible improvement' over Opus 4.7 (simonwillison.net/2026/May/28/claude-opus-4-8/). 4x less likely to miss code flaws vs predecessor.

Agent: Top closed agent with SWE-Pro 69.2 and new 'dynamic workflow' tooling (Claude Code 2.1.154). Powers production Hermes / Open-Claude setups. TB2.1 74.6 per Anthropic self-report — TB2.0 leaderboard not yet updated. Fast mode at $10/$50 reduces agentic cost vs standard rate.

★

Our picks for this build

Sourced from the State of Local AI snapshot — the model + quant + backend we'd actually deploy on this hardware today, with the recipe in the setup guide below.

Best dense

Qwen 3.6 27B (dense)

27 B Apache 2.0 🤗

Apr 22 2026. Dense 27B that hits 77.2% SWE-Bench Verified — beats much larger MoEs on coding. Vision-capable, 262 K native context. Best single-24 GB-card coder right now.

≥20 GB Q4 250 t/s

HLE24.0%
TB259.3%
SWE-Pro53.5%
SWE-Ver77.2%

Coding: The new local-coding king under 200B on r/LocalLLaMA — matches Claude Opus 4.5 on TB2 per Qwen's launch claims, beats Qwen3.5-397B-A17B on every coding eval. Daily-driver pick for Cline at Q4_K_M on a single Pro 6000 or M3 Ultra. Confirmed running ~160 tok/s with MTP on RTX 6000 per dzombak.com vLLM recipe.

Agent: Genuinely useful in Open-Claude / Claude Code routing — community reports 30-min+ sessions completing without derail. Still trails closed frontier on the very longest loops. Caps at agents:3 per site rule (sub-200B, TB2 59.3 below 65% threshold).

Best MoE that fits

Qwen 3.6 35B-A3B (MoE)

35 B · 3B active Apache 2.0 🤗

Apr 2026 release. 35B / 3B active MoE — beats Gemma 4-31B on agentic coding, matches Sonnet on most vision tasks. Native 262 K context (extensible to 1 M), ~18 GB at Q4. The new local-coding king under 200 B.

≥22 GB Q4 750 t/s

HLE21.4%
TB251.5%
SWE-Pro49.5%
SWE-Ver73.4%

Coding: r/LocalLLaMA's pick for fast local coding on a 24 GB card at Q4_K_M — 3B active so it's snappy. Vibes-codes 'perfectly fine' in OpenCode/Claude Code per multiple weekly-megathreads. Simon Willison's pelican test (April 2026): 'Qwen3.6-35B-A3B on my laptop drew me a better pelican than Claude Opus 4.7' — still resonating in the community.

Agent: Solid in 5-15 tool-hop loops in Cline. Long-horizon (60+ min) Open-Claude sessions still lose thread — 3B active is a ceiling on planning. Note: Qwen-self-reported TB2 51.5 vs community 23-24% — gap is harness-driven (Terminus-2 vs little-coder agent).

Dense runner-up

Mistral Medium 3.5 128B

128 B Modified MIT 🤗

Apr 30 2026. Western 128B dense with vision + 256 K context. 77.6% SWE-Bench Verified; first credible mid-tier open-weight from Mistral in months. Modified MIT.

≥80 GB Q4 93 t/s

SWE-Ver77.6%

Coding: Apr 30 2026 launch with built-in PR-opening coding agent. Western 128B-dense with vision + 256K — early r/LocalLLaMA reports treat it as a credible Cline driver but trailing Qwen 3.6-27B on real refactors.

Agent: Mistral's agent SDK is OK; in Open-Claude it handles ~20-min sessions reliably. Long-horizon ceiling still unclear pending community evals.

Every open-weights model that fits, ranked by composite score

Composite blends benchmark averages (60 %) with editorial 0-5 ratings (40 %). Closed-frontier references mix into the ranking and stay amber-tinted.

Model↕	tg/s↕	pp↕	TTFT @ 100K↕	HLE↕	TB2↕	SWE-Pro↕	SWE-Ver↕	Aider↕	LCB↕	GPQA↕	MMLU-Pro↕	Score↕
Qwen 3 235B-A22B (MoE)235 B · 22 B active · moe🤗	260 t/s	21,000 pp	7.0 s	—	—	—	—	—	70.7%	81.1%	84.4%	4964
Qwen 3 Next 80B-A3B (MoE)80 B · 3 B active · moe🤗	—	44,000 pp	3.3 s	—	—	—	—	—	78.4%	77.2%	82.7%	4920
Mistral Medium 3.5 128B128 B · dense🤗	93 t/s	13,500 pp	10 s	—	—	—	77.6%	—	—	—	—	4689
GLM-5.1 754B (MoE)754 B · 40 B active · moe🤗	88 t/s	6,200 pp	28 s	52.3%	63.5%	58.4%	77.8%	—	84.1%	86.2%	91.7%	4630
Qwen 3.5 9B9 B · dense🤗	409 t/s	110,000 pp	1.3 s	—	—	—	—	—	65.6%	81.7%	82.5%	4623
DeepSeek V4-Pro 1.6T (MoE)1600 B · 49 B active · moe🤗	36 t/s	3,900 pp	45 s	37.7%	67.9%	55.4%	80.6%	71.6%	93.5%	90.1%	87.5%	4519
GLM-4.5-Air 106B (MoE)106 B · 12 B active · moe🤗	—	44,000 pp	3.3 s	—	—	—	57.6%	—	—	—	—	4488
Kimi K2.6 1T (MoE)1000 B · 32 B active · moe🤗	55 t/s	3,900 pp	45 s	35.9%	66.7%	58.6%	80.2%	—	89.6%	90.5%	—	4479
DeepSeek V4-Flash 284B (MoE)284 B · 13 B active · moe🤗	440 t/s	21,000 pp	7.0 s	29.4%	56.9%	52.6%	79.0%	—	91.6%	88.1%	86.4%	4379
NVIDIA Nemotron 3 Ultra 550B-A55B (MoE)550 B · 55 B active · moe🤗	104 t/s	21,000 pp	7.0 s	26.7%	—	—	71.9%	—	89.0%	87.0%	86.8%	4372
DiffusionGemma 26B-A4B26 B · 4 B active · diffusion-moe🤗	750 t/s	31,000 pp	4.5 s	—	—	—	—	—	69.1%	73.2%	77.6%	4358
Mistral Small 4 119B-A6B (MoE)119 B · 6 B active · moe🤗	—	44,000 pp	3.3 s	—	—	—	—	—	—	71.2%	—	4301
Qwen 3.5 397B-A17B (MoE)397 B · 17 B active · moe🤗	336 t/s	21,000 pp	7.0 s	37.6%	52.5%	50.9%	76.4%	—	83.6%	88.4%	87.8%	4300
Qwen 3.6 27B (dense)27 B · dense🤗	250 t/s	31,000 pp	4.5 s	24.0%	59.3%	53.5%	77.2%	—	83.9%	87.8%	86.2%	4280
Qwen 3 32B32 B · dense🤗	211 t/s	31,000 pp	4.5 s	—	—	—	—	—	—	65.7%	65.5%	4278
Kimi K2.5 1T (MoE)1000 B · 32 B active · moe🤗	55 t/s	3,900 pp	45 s	30.1%	43.2%	50.7%	76.8%	—	85.0%	87.6%	87.1%	4213
GLM-5.2 753B (MoE)753 B · 39 B active · moe🤗	90 t/s	6,200 pp	28 s	54.7%	—	62.1%	—	—	—	91.2%	—	4200
DeepSeek R1 Distill 70B70 B · dense🤗	170 t/s	13,500 pp	10 s	—	—	—	—	—	57.5%	65.2%	84.0%	4171
Phi-4 14B14 B · dense🤗	340 t/s	64,000 pp	2.2 s	—	—	—	—	—	—	56.1%	70.4%	4160
DeepSeek V3 671B (MoE)671 B · 37 B active · moe🤗	95 t/s	6,200 pp	28 s	22.2%	39.6%	—	—	74.2%	89.6%	79.9%	85.0%	4151
Qwen 3.6 35B-A3B (MoE)35 B · 3 B active · moe🤗	750 t/s	31,000 pp	4.5 s	21.4%	51.5%	49.5%	73.4%	—	80.4%	86.0%	85.2%	4084
Qwen 3.5 122B-A10B (MoE)122 B · 10 B active · moe🤗	—	44,000 pp	3.3 s	25.3%	49.4%	—	72.0%	—	78.9%	86.6%	86.7%	4021
Tencent Hy3 295B-A21B (MoE)295 B · 21 B active · moe🤗	272 t/s	21,000 pp	7.0 s	25.5%	54.4%	57.9%	78.0%	—	—	90.4%	87.4%	3971
Llama 4 Maverick 400B-A17B (MoE)400 B · 17 B active · moe🤗	336 t/s	21,000 pp	7.0 s	—	—	—	—	—	43.4%	69.8%	80.5%	3905
NVIDIA Nemotron 3 Super 120B-A12B (MoE)120 B · 12 B active · moe🤗	—	44,000 pp	3.3 s	18.3%	31.0%	—	60.5%	—	81.2%	79.2%	83.7%	3839
Llama 3.1 405B405 B · dense🤗	78 t/s	21,000 pp	7.0 s	—	—	—	—	—	—	51.1%	73.4%	3762
Gemma 4 12B Unified (dense)12 B · dense🤗	397 t/s	64,000 pp	2.2 s	5.2%	—	—	—	—	72.0%	78.8%	77.2%	3759
Gemma 4 31B (dense)31 B · dense🤗	218 t/s	31,000 pp	4.5 s	19.5%	42.9%	35.7%	52.0%	—	80.0%	84.3%	85.2%	3697
Xiaomi MiMo V2.5 310B-A15B (MoE)310 B · 15 B active · moe🤗	381 t/s	21,000 pp	7.0 s	—	65.8%	56.1%	—	—	—	—	—	3690
MiniMax M2.7 230B-A10B (MoE)230 B · 10 B active · moe🤗	572 t/s	21,000 pp	7.0 s	—	45.1%	56.2%	78.0%	—	—	—	—	3620
Xiaomi MiMo V2.5-Pro 1T-A42B (MoE)1020 B · 42 B active · moe🤗	42 t/s	3,900 pp	45 s	34.0%	68.4%	57.2%	78.9%	—	—	—	—	3614
MiniMax M3 428B-A23B (MoE)428 B · 23 B active · moe🤗	249 t/s	21,000 pp	7.0 s	—	—	59.0%	—	—	—	—	—	3579
GPT-OSS 120B120 B · 5 B active · moe🤗	—	44,000 pp	3.3 s	18.5%	18.7%	16.2%	62.4%	—	87.8%	80.9%	90.0%	3573
Kimi K2 1T (MoE)1000 B · 32 B active · moe🤗	55 t/s	3,900 pp	45 s	4.7%	27.8%	—	65.8%	60.0%	85.3%	75.1%	81.1%	3408
Mistral Small 3 24B24 B · dense🤗	281 t/s	31,000 pp	4.5 s	—	—	—	—	—	—	45.3%	66.0%	3361
Gemma 3 27B27 B · dense🤗	250 t/s	31,000 pp	4.5 s	—	—	—	—	—	—	42.4%	67.5%	3321
Llama 4 Scout 109B-A17B (MoE)109 B · 17 B active · moe🤗	—	44,000 pp	3.3 s	—	—	—	—	—	32.8%	57.2%	74.3%	3315
Devstral 2 123B (dense)123 B · dense🤗	97 t/s	13,500 pp	10 s	—	32.6%	—	72.2%	—	—	—	—	3174
Gemma 4 26B-A4B (MoE)26 B · 4 B active · moe🤗	750 t/s	31,000 pp	4.5 s	8.7%	34.2%	13.8%	17.4%	—	77.1%	82.3%	82.6%	3060
Qwen 3 Coder 30B-A3B (MoE)30 B · 3 B active · moe🤗	750 t/s	31,000 pp	4.5 s	—	—	—	50.3%	—	—	—	—	3042
Llama 3.3 70B70 B · dense🤗	170 t/s	13,500 pp	10 s	—	—	—	—	—	28.8%	50.5%	68.9%	2990
Llama 3.1 8B8 B · dense🤗	460 t/s	110,000 pp	1.3 s	—	—	—	—	—	—	34.6%	49.0%	2527
Qwen 3 8B8 B · dense🤗	460 t/s	110,000 pp	1.3 s	2.8%	—	—	—	—	—	47.0%	65.5%	2325
Kimi K2.7 Code 1T (MoE)1000 B · 32 B active · moe🤗	55 t/s	3,900 pp	45 s	—	—	—	—	—	—	—	—	48
StepFun Step 3.5 Flash 197B-A11B (MoE)197 B · 11 B active · moe🤗	—	44,000 pp	3.3 s	—	—	—	—	—	—	—	—	44
Mistral Large 3 675B-A41B (MoE)675 B · 41 B active · moe🤗	86 t/s	6,200 pp	28 s	—	—	—	—	—	—	—	—	42
Cohere Command A+ 218B-A25B (MoE)218 B · 25 B active · moe🤗	229 t/s	21,000 pp	7.0 s	—	—	—	—	—	—	—	—	39
Gemini 3.1 ProGoogle DeepMind · closed	125 t/s	—	2.1 min	44.7%	80.2%	54.2%	80.6%	—	91.7%	94.3%	91.0%	—
ChatGPT 5.5OpenAI · closed	61 t/s	—	1.6 min	52.2%	82.0%	58.6%	88.7%	88.0%	—	93.6%	—	—
Claude Sonnet 5Anthropic · closed	—	—	—	57.4%	80.4%	63.2%	—	—	—	—	—	—
Claude Opus 4.8Anthropic · closed	59 t/s	—	2.9 min	57.9%	—	69.2%	88.6%	—	—	93.6%	—	—

Open in the live picker (Q2 / Q4 / Q5 / Q8 toggles) → Compare 12× RTX Pro 6000 Blackwell rack (1152 GB) with another build → Try other hardware → Submit a benchmark for 12× RTX Pro 6000 Blackwell rack (1152 GB) ↗

Last updated 2026-07-11