Best AI models that run on DGX H200 — 8× H200 server (1.13 TB HBM3e)

The 2025-26 reference frontier-LLM inference box.

NVIDIA · 8U DGX / HGX server rack

DGX H200 — 8× H200 server (1.13 TB HBM3e)

1128 GB 1100 GB usable 4800 GB/s $380k

switch in the live picker →

📺 Reviews of DGX H200 — 8× H200 server (1.13 TB HBM3e)

▶ Inside the SUPER NVIDIA H200 Server From Supermicro (8U HGX H200)

NVIDIA DGX H200 SuperMicro HGX Lambda Labs

What models fit this build

DGX H200 — 8× H200 server (1.13 TB HBM3e) has 1100 GB of usable memory. Here's which open-weights model sizes fit at each quant, at ~8K context with an FP8 KV cache.

Largest comfortable fit: 1T+ MoE at Q8 (~1063 GB).

Model size	Q2	Q4	Q5	Q8
7–8B	Fits	Fits	Fits	Fits
13–14B	Fits	Fits	Fits	Fits
30–32B	Fits	Fits	Fits	Fits
70–72B	Fits	Fits	Fits	Fits
~120B MoE	Fits	Fits	Fits	Fits
~235B MoE	Fits	Fits	Fits	Fits
~670B MoE	Fits	Fits	Fits	Fits
1T+ MoE	Fits	Fits	Fits	Fits

✓ = the weights + FP8 KV cache fit within this build's usable memory at ~8K context. Longer context needs more — size any model in the picker →

Closed-frontier reference (May 2026)

What "API-grade" actually scores right now. Use this as the ceiling — anything local will lag here by some margin, and that's fine for most workflows.

Google DeepMind 2026-02

Gemini 3.1 Pro

2.0M ctx $2 / $12 per M tok

API: 125 t/s out · +30 s thinking

HLE44.7%
Terminal-Bench 280.2%
SWE-Bench Pro54.2%
SWE-Bench Ver80.6%
Aider Polyglot—
LiveCodeBench91.7%
GPQA Diamond94.3%
MMLU-Pro91.0%

Coding: Coding-strong, especially repo-level work with 2M context. Less of a Cursor/Cline default than Claude — Gemini Code Assist users prefer it. Top of leaderboards on GPQA + LCB. Now #7 on OpenRouter coding by volume (120B tokens) — Gemini 3.5 Flash will displace it next month.

Agent: TB2 80.2 makes it agent-grade. Used widely on Google's Agent Builder; less common in Open-Claude/Hermes. Reliable in long Vertex-AI Agent runs (multi-hour).

OpenAI 2026-04

ChatGPT 5.5

922K ctx $5 / $30 per M tok

API: 61 t/s out · +28 s thinking

HLE52.2%
Terminal-Bench 282.0%
SWE-Bench Pro58.6%
SWE-Bench Ver88.7%
Aider Polyglot88.0%
LiveCodeBench—
GPQA Diamond93.6%
MMLU-Pro—

Coding: Codex CLI + GPT-5.5 is the top of Terminal-Bench. r/ChatGPTCoding has shifted to it for daily coding; Cursor users mixed (some prefer Claude Sonnet 4.6 for diff quality).

Agent: Strongest published agent score (TB2 82.0%, re-verified 2026-05-14). Widely used in OpenAI Assistants, AutoGPT-style swarms, and Open-Claude routing. Reliable in 4-8h autonomous sessions.

Anthropic 2026-06

Claude Sonnet 5

1.0M ctx $3 / $15 per M tok

HLE57.4%
Terminal-Bench 280.4%
SWE-Bench Pro63.2%
SWE-Bench Ver—
Aider Polyglot—
LiveCodeBench—
GPQA Diamond—
MMLU-Pro—

Coding: Anthropic's 2026-06-30 release: near-Opus intelligence at the old Sonnet price. Terminal-Bench 2.1 jumps to 80.4 (from Sonnet 4.6's 53.4), putting it in the top tier of agentic coders next to GPT-5.5 (82) and Gemini 3.1 Pro (80.2). 1M context, adaptive thinking on by default; the daily workhorse for Cursor / Zed / Cline users who want closed-model diff quality.

Agent: TB2 80.4 makes Sonnet 5 a top-tier agent driver (Sonnet 4.6 was mid-tier at 53.4). SWE-Bench Pro 63.2 lands between GPT-5.5 (58.6) and Opus 4.7 (69.2). Anthropic's tool-use SDK keeps it the most reliable closed model for hand-rolled agent loops.

Anthropic 2026-05

Claude Opus 4.8

1.0M ctx $5 / $25 per M tok

API: 59 t/s out · +22 s thinking

HLE57.9%
Terminal-Bench 2—
SWE-Bench Pro69.2%
SWE-Bench Ver88.6%
Aider Polyglot—
LiveCodeBench—
GPQA Diamond93.6%
MMLU-Pro—

Coding: Cursor / Cline / Zed power-user pick when budget allows. SWE-Pro 69.2 and TB2.1 74.6 say it all — best closed model for real software engineering. Simon Willison describes it as 'a modest but tangible improvement' over Opus 4.7 (simonwillison.net/2026/May/28/claude-opus-4-8/). 4x less likely to miss code flaws vs predecessor.

Agent: Top closed agent with SWE-Pro 69.2 and new 'dynamic workflow' tooling (Claude Code 2.1.154). Powers production Hermes / Open-Claude setups. TB2.1 74.6 per Anthropic self-report — TB2.0 leaderboard not yet updated. Fast mode at $10/$50 reduces agentic cost vs standard rate.

★

Our picks for this build

Sourced from the State of Local AI snapshot — the model + quant + backend we'd actually deploy on this hardware today, with the recipe in the setup guide below.

Recommended

Kimi K2.7 Code 1T (MoE)

1000 B · 32B active Modified MIT 🤗

Jun 2026 release — code-specialized variant built on K2.6 with ~30% fewer thinking tokens for the same task. Same 1 T total / 32 B active MoE (384 experts top-8 + 1 shared), 256 K context, Modified MIT. Internal Moonshot benchmarks only (Kimi Code Bench v2 62.0, MCP Atlas 76.0, MCP Mark Verified 81.1) — standard SWE-Bench / TB2 cells stay null until third-party leaderboards land.

≥600 GB Q4 85 t/s

No published benchmarks yet — see model card.

Coding: Moonshot frames it as 'code-first K2.6 with shorter thinking chains' — claims ~30 % fewer thinking tokens for the same long-horizon coding task. Worth watching for r/LocalLLaMA Cline / Aider thread reactions once they land; vendor benchmarks only at launch.

Agent: MCP Atlas 76.0 and MCP Mark Verified 81.1 are vendor-reported MCP-specific scores — agentic positioning, but no Terminal-Bench or SWE-Bench cross-check yet. Treat as 'promising, unverified outside Moonshot' until tbench.ai or another public leaderboard picks it up.

Also runs well

NVIDIA Nemotron 3 Ultra 550B-A55B (MoE)

550 B · 55B active OpenMDW-1.1 🤗

Jun 2026. NVIDIA's frontier hybrid Mamba-2 + LatentMoE + attention with MTP — 55 B active / 550 B total, native 1 M ctx (RULER@1M 94.7). SWE-V 71.9, LCB v6 89.0, GPQA 87.0, HLE 26.7. OpenMDW-1.1 (commercial OK).

≥340 GB Q4 62 t/s

HLE26.7%
SWE-Ver71.9%
LCB89.0%

Coding: Day-0 vLLM + SGLang + TRT-LLM support with MTP-5 speculative decoding; LCB v6 89.0 sits alongside V4-Pro / Kimi K2.6. 8× H100/B200 minimum, so deployment is data-center class.

Agent: SWE-V 71.9 is below the agents:5 cluster (V4-Pro / K2.6 at 80+) and TB2.0 isn't yet on the leaderboard; caps at agents:4 until tbench.ai picks it up.

Setup guide for DGX H200 — 8× H200 server (1.13 TB HBM3e)

Every known recipe for running a model on this build — sourced from the State of Local AI 2026-07-05 snapshot (2026-07-05). Pick the one that matches your model + quant, then follow the linked original write-up.

Recipes for DGX H200 — 8× H200 server (1.13 TB HBM3e)
Model	Decode tok/s	Prompt processing	Recipe	Runs
kimi-k2-7-code-1t-moe@ native INT4 on vLLM (TP=8, expert-parallel)	—	—	vLLM Recipes (official K2.7 Code command, 8xH200 INT4); tok/s = K2.6 same-box SGLang INT4 (paxsaroffcuts) ↗
nemotron-3-ultra-550b-a55b-moe@ NVFP4 + FP8 KV on Dynamo + vLLM (TP=8, expert-parallel, MTP)	batch~58.7@ 4K (10-conc. aggregate)	—	NVIDIA ai-dynamo/dynamo recipes (8xH200 TP8+EP, NVFP4+FP8, MTP) ↗

Want to compare this against other builds? Open the live picker (Q2 / Q4 / Q5 / Q8 toggles) or see best build by budget.

See all recipes

Every open-weights model that fits, ranked by composite score

Composite blends benchmark averages (60 %) with editorial 0-5 ratings (40 %). Closed-frontier references mix into the ranking and stay amber-tinted.

Model↕	tg/s↕	pp↕	TTFT @ 100K↕	HLE↕	TB2↕	SWE-Pro↕	SWE-Ver↕	Aider↕	LCB↕	GPQA↕	MMLU-Pro↕	Score↕
Qwen 3 235B-A22B (MoE)235 B · 22 B active · moe🤗	155 t/s	11,000 pp	11 s	—	—	—	—	—	70.7%	81.1%	84.4%	4964
Qwen 3 Next 80B-A3B (MoE)80 B · 3 B active · moe🤗	517 t/s	23,000 pp	5.0 s	—	—	—	—	—	78.4%	77.2%	82.7%	4920
Mistral Medium 3.5 128B128 B · dense🤗	93 t/s	7,100 pp	16 s	—	—	—	77.6%	—	—	—	—	4689
GLM-5.1 754B (MoE)754 B · 40 B active · moe🤗	93 t/s	3,700 pp	35 s	52.3%	63.5%	58.4%	77.8%	—	84.1%	86.2%	91.7%	4630
Qwen 3.5 9B9 B · dense🤗	498 t/s	58,000 pp	2.5 s	—	—	—	—	—	65.6%	81.7%	82.5%	4623
DeepSeek V4-Pro 1.6T (MoE)1600 B · 49 B active · moe🤗	56 t/s	2,300 pp	55 s	37.7%	67.9%	55.4%	80.6%	71.6%	93.5%	90.1%	87.5%	4519
GLM-4.5-Air 106B (MoE)106 B · 12 B active · moe🤗	129 t/s	23,000 pp	5.0 s	—	—	—	57.6%	—	—	—	—	4488
Kimi K2.6 1T (MoE)1000 B · 32 B active · moe🤗	85 t/s	2,300 pp	55 s	35.9%	66.7%	58.6%	80.2%	—	89.6%	90.5%	—	4479
DeepSeek V4-Flash 284B (MoE)284 B · 13 B active · moe🤗	262 t/s	11,000 pp	11 s	29.4%	56.9%	52.6%	79.0%	—	91.6%	88.1%	86.4%	4379
NVIDIA Nemotron 3 Ultra 550B-A55B (MoE)550 B · 55 B active · moe🤗	62 t/s	11,000 pp	11 s	26.7%	—	—	71.9%	—	89.0%	87.0%	86.8%	4372
DiffusionGemma 26B-A4B26 B · 4 B active · diffusion-moe🤗	750 t/s	15,800 pp	6.5 s	—	—	—	—	—	69.1%	73.2%	77.6%	4358
Mistral Small 4 119B-A6B (MoE)119 B · 6 B active · moe🤗	258 t/s	23,000 pp	5.0 s	—	—	—	—	—	—	71.2%	—	4301
Qwen 3.5 397B-A17B (MoE)397 B · 17 B active · moe🤗	201 t/s	11,000 pp	11 s	37.6%	52.5%	50.9%	76.4%	—	83.6%	88.4%	87.8%	4300
Qwen 3.6 27B (dense)27 B · dense🤗	250 t/s	15,800 pp	6.5 s	24.0%	59.3%	53.5%	77.2%	—	83.9%	87.8%	86.2%	4280
Qwen 3 32B32 B · dense🤗	211 t/s	15,800 pp	6.5 s	—	—	—	—	—	—	65.7%	65.5%	4278
Kimi K2.5 1T (MoE)1000 B · 32 B active · moe🤗	85 t/s	2,300 pp	55 s	30.1%	43.2%	50.7%	76.8%	—	85.0%	87.6%	87.1%	4213
GLM-5.2 753B (MoE)753 B · 39 B active · moe🤗	95 t/s	3,700 pp	35 s	54.7%	—	62.1%	—	—	—	91.2%	—	4200
DeepSeek R1 Distill 70B70 B · dense🤗	170 t/s	7,100 pp	16 s	—	—	—	—	—	57.5%	65.2%	84.0%	4171
Phi-4 14B14 B · dense🤗	390 t/s	33,500 pp	3.6 s	—	—	—	—	—	—	56.1%	70.4%	4160
DeepSeek V3 671B (MoE)671 B · 37 B active · moe🤗	100 t/s	3,700 pp	35 s	22.2%	39.6%	—	—	74.2%	89.6%	79.9%	85.0%	4151
Qwen 3.6 35B-A3B (MoE)35 B · 3 B active · moe🤗	750 t/s	15,800 pp	6.5 s	21.4%	51.5%	49.5%	73.4%	—	80.4%	86.0%	85.2%	4084
Qwen 3.5 122B-A10B (MoE)122 B · 10 B active · moe🤗	155 t/s	23,000 pp	5.0 s	25.3%	49.4%	—	72.0%	—	78.9%	86.6%	86.7%	4021
Tencent Hy3 295B-A21B (MoE)295 B · 21 B active · moe🤗	162 t/s	11,000 pp	11 s	25.5%	54.4%	57.9%	78.0%	—	—	90.4%	87.4%	3971
Llama 4 Maverick 400B-A17B (MoE)400 B · 17 B active · moe🤗	201 t/s	11,000 pp	11 s	—	—	—	—	—	43.4%	69.8%	80.5%	3905
NVIDIA Nemotron 3 Super 120B-A12B (MoE)120 B · 12 B active · moe🤗	129 t/s	23,000 pp	5.0 s	18.3%	31.0%	—	60.5%	—	81.2%	79.2%	83.7%	3839
Llama 3.1 405B405 B · dense🤗	47 t/s	11,000 pp	11 s	—	—	—	—	—	—	51.1%	73.4%	3762
Gemma 4 12B Unified (dense)12 B · dense🤗	455 t/s	33,500 pp	3.6 s	5.2%	—	—	—	—	72.0%	78.8%	77.2%	3759
Gemma 4 31B (dense)31 B · dense🤗	218 t/s	15,800 pp	6.5 s	19.5%	42.9%	35.7%	52.0%	—	80.0%	84.3%	85.2%	3697
Xiaomi MiMo V2.5 310B-A15B (MoE)310 B · 15 B active · moe🤗	227 t/s	11,000 pp	11 s	—	65.8%	56.1%	—	—	—	—	—	3690
MiniMax M2.7 230B-A10B (MoE)230 B · 10 B active · moe🤗	341 t/s	11,000 pp	11 s	—	45.1%	56.2%	78.0%	—	—	—	—	3620
Xiaomi MiMo V2.5-Pro 1T-A42B (MoE)1020 B · 42 B active · moe🤗	65 t/s	2,300 pp	55 s	34.0%	68.4%	57.2%	78.9%	—	—	—	—	3614
MiniMax M3 428B-A23B (MoE)428 B · 23 B active · moe🤗	148 t/s	11,000 pp	11 s	—	—	59.0%	—	—	—	—	—	3579
GPT-OSS 120B120 B · 5 B active · moe🤗	310 t/s	23,000 pp	5.0 s	18.5%	18.7%	16.2%	62.4%	—	87.8%	80.9%	90.0%	3573
Kimi K2 1T (MoE)1000 B · 32 B active · moe🤗	85 t/s	2,300 pp	55 s	4.7%	27.8%	—	65.8%	60.0%	85.3%	75.1%	81.1%	3408
Mistral Small 3 24B24 B · dense🤗	281 t/s	15,800 pp	6.5 s	—	—	—	—	—	—	45.3%	66.0%	3361
Gemma 3 27B27 B · dense🤗	250 t/s	15,800 pp	6.5 s	—	—	—	—	—	—	42.4%	67.5%	3321
Llama 4 Scout 109B-A17B (MoE)109 B · 17 B active · moe🤗	93 t/s	23,000 pp	5.0 s	—	—	—	—	—	32.8%	57.2%	74.3%	3315
Devstral 2 123B (dense)123 B · dense🤗	97 t/s	7,100 pp	16 s	—	32.6%	—	72.2%	—	—	—	—	3174
Gemma 4 26B-A4B (MoE)26 B · 4 B active · moe🤗	750 t/s	15,800 pp	6.5 s	8.7%	34.2%	13.8%	17.4%	—	77.1%	82.3%	82.6%	3060
Qwen 3 Coder 30B-A3B (MoE)30 B · 3 B active · moe🤗	750 t/s	15,800 pp	6.5 s	—	—	—	50.3%	—	—	—	—	3042
Llama 3.3 70B70 B · dense🤗	170 t/s	7,100 pp	16 s	—	—	—	—	—	28.8%	50.5%	68.9%	2990
Llama 3.1 8B8 B · dense🤗	560 t/s	58,000 pp	2.5 s	—	—	—	—	—	—	34.6%	49.0%	2527
Qwen 3 8B8 B · dense🤗	560 t/s	58,000 pp	2.5 s	2.8%	—	—	—	—	—	47.0%	65.5%	2325
Kimi K2.7 Code 1T (MoE)1000 B · 32 B active · moe🤗	85 t/s	2,300 pp	55 s	—	—	—	—	—	—	—	—	48
StepFun Step 3.5 Flash 197B-A11B (MoE)197 B · 11 B active · moe🤗	141 t/s	23,000 pp	5.0 s	—	—	—	—	—	—	—	—	44
Mistral Large 3 675B-A41B (MoE)675 B · 41 B active · moe🤗	90 t/s	3,700 pp	35 s	—	—	—	—	—	—	—	—	42
Cohere Command A+ 218B-A25B (MoE)218 B · 25 B active · moe🤗	136 t/s	11,000 pp	11 s	—	—	—	—	—	—	—	—	39
Gemini 3.1 ProGoogle DeepMind · closed	125 t/s	—	2.1 min	44.7%	80.2%	54.2%	80.6%	—	91.7%	94.3%	91.0%	—
ChatGPT 5.5OpenAI · closed	61 t/s	—	1.6 min	52.2%	82.0%	58.6%	88.7%	88.0%	—	93.6%	—	—
Claude Sonnet 5Anthropic · closed	—	—	—	57.4%	80.4%	63.2%	—	—	—	—	—	—
Claude Opus 4.8Anthropic · closed	59 t/s	—	2.9 min	57.9%	—	69.2%	88.6%	—	—	93.6%	—	—

Open in the live picker (Q2 / Q4 / Q5 / Q8 toggles) → Compare DGX H200 — 8× H200 server (1.13 TB HBM3e) with another build → Try other hardware → Submit a benchmark for DGX H200 — 8× H200 server (1.13 TB HBM3e) ↗

Last updated 2026-07-11