Multi-token prediction (MTP) merged into llama.cpp on May 16 via PR #22673 by am17an. The feature is shipping in beta — feature-flagged behind --mtp on llama-cli and llama-server, off by default.
A community benchmark posted within the first day after merge reported Qwen 3.6-27B decoding at 65 tok/s on a single RTX 5090, up from 38 tok/s without MTP — a 1.71× speedup at Q4_K_M with no perceptible quality regression in side-by-side prompt comparisons. That single data point is what we have so far; broader build × model coverage will take a couple of refresh cycles to land.
What MTP actually is
The model is trained with extra output heads that predict the next 2, 3, or 4 tokens simultaneously off the same shared backbone hidden state. The main head then re-runs to verify the candidate sequence. If the speculative tokens match, you get them effectively for free. If not, you fall back to single-token decoding for that step.
This collapses what speculative decoding traditionally does — running a small draft model alongside a big target model — into a single model with built-in prediction heads. The overhead is roughly one additional transformer layer per extra prediction position, not a full second network. Memory cost is small; throughput cost during failed verifications is the main downside, but on models trained for MTP the hit rate is high enough that the average decode is materially faster.
What this requires of the model
MTP needs prediction heads baked into training — the runtime can’t synthesize them after the fact. The cited benchmark used Qwen 3.6-27B at Q4_K_M; the --mtp flag is a no-op for any model whose checkpoint lacks the extra head weights, so leaving the flag on by default doesn’t hurt.
DeepSeek-V3’s paper popularised MTP as a training-time auxiliary objective. Which open-weights models that shipped after the DeepSeek-V3 paper carry the heads in their public checkpoints — and how many positions ahead — is something we’ll catalogue per-model in the next data refresh, citing each model card directly rather than going on architectural rumour.
Caveats
- Beta means beta. PR #22673 itself flags this; expect rough edges and watch the linked llama.cpp issue tracker before depending on it in production.
- Speedups scale with what’s bottlenecking decode. On compute-rich, bandwidth-fed rigs the speculative pass has headroom; on bandwidth-bound rigs the win shrinks. The 1.71× on a 5090 is one point on that gradient — don’t assume it’s universal.
How to enable
Build llama.cpp off main after May 16 and pass --mtp to llama-server or llama-cli. Upstream support in Ollama, LM Studio, and Jan typically lags llama.cpp main by a release cycle — check each runtime’s release notes for the first build that picks it up.
Sources
- llama.cpp PR #22673 — MTP Support (am17an, merged 2026-05-16)
- Community gist: running Qwen 3.6 with MTP (38 → 65 tok/s on RTX 5090 benchmark)
- Multi-token prediction tutorial — DataCamp