Diffusion Language Models · Inference Optimization

Almost every LLM you use is autoregressive (AR): it emits one token per forward pass, left to right. Diffusion language models take a different path…

Almost every LLM you use is autoregressive (AR): it emits one token per forward pass, left to right. Diffusion language models take a different path borrowed from image generation: start from a fully masked sequence and iteratively denoise all positions in parallel over a handful of steps. The payoff is throughput, reported figures over 1,000 tokens per second, an order of magnitude above speed-optimized AR models. The cost is some quality, and the maturity is early. This chapter explains how they work, where they win, and what an engineer should actually do with them in 2026.

How They Work
The Speed Advantage and the Tradeoff
The 2026 Landscape
Hybrids: Draft with Diffusion, Verify with AR
Where Diffusion Wins and Where AR Still Wins
Maturity and What to Do Today
Interview Questions
References

How They Work

An AR model factorizes a sequence left to right: each token is predicted from the past only, one per forward pass. A masked diffusion model instead learns to reconstruct a whole sequence from a corrupted (masked) version, refining all positions at once over a few denoising steps, with bidirectional attention (each position sees both past and future).

The canonical reference is LLaDA (arXiv:2502.09992). The mechanism, at a teachable level:

Forward process: start from clean text; at noise level t, replace each token with a [MASK] with probability t. At t=1 the sequence is fully masked.
Reverse process: a Transformer (a "mask predictor") is trained to predict the original token at every masked position simultaneously, optimizing a likelihood lower bound (not the exact likelihood AR models use).
Sampling: begin fully masked and denoise from t=1 to 0. At each step the model predicts all masked tokens, and an unmasking schedule decides how many of the highest-confidence predictions to commit and how many to re-mask for the next step. Fewer steps means faster and lower quality; one token per step approaches AR quality but forfeits the speed.

Parallel decoding and bidirectional context fall out of this directly, and they give diffusion a notable trick: LLaDA reports beating GPT-4o on a reversal task (completing a poem backwards), because the objective is not locked to left-to-right order.

The cleanest mental model is not "AR vs diffusion" but an interpolation between them, established by Block Diffusion (BD3-LMs, arXiv:2503.09573): split the sequence into blocks, go AR across blocks and diffusion within a block. This recovers two things pure diffusion struggles with, KV caching and variable-length output, and the block size is the knob (block size 1 is AR, block size equals the whole sequence is pure diffusion). Almost every fast 2026 system ships some block or semi-AR variant.

The Speed Advantage and the Tradeoff

The speedup comes from one place: tokens committed per forward pass. AR emits one; diffusion commits many over a few steps, so wall-clock throughput rises when the model can confidently unmask several positions at once. It is a latency and throughput win, not inherently a quality win.

Reported throughput (all vendor or benchmark claims, conditions vary): commercial diffusion models report on the order of 700 to 1,500 tokens per second on a single datacenter GPU, which their makers frame as several times faster than speed-optimized AR baselines (up to ~10x for the best-case code-completion path, closer to ~5x for reasoning workloads). Independent measurement of one commercial model put it around 1,100 tokens per second.

The tradeoff is intrinsic to the unmasking schedule: committing more tokens per step is faster but can produce locally incoherent output, so the practical best quality comes from unmasking fewer tokens per step, which slows things down. Where the quality gap stands in 2026, as reported:

On code, the gap is small. Diffusion models roughly match similar-tier AR models on code-completion benchmarks; code is the consensus sweet spot.
On knowledge and hard reasoning, the gap is larger, and no diffusion model is frontier-class. A recognized theoretical headwind is that masked diffusion optimizes a likelihood bound, not exact likelihood.

The 2026 Landscape

Treat versions, speeds, and prices as a perishable snapshot.

System	Type	Notes (reported)
Mercury / Mercury 2 (Inception Labs)	Closed, code-focused, then general	The clearest production option: OpenAI-compatible API and a cloud-marketplace listing, ~1,000-1,200 tokens/sec, public pricing. Marketed as a fast reasoning model, a claim some analysts dispute given diffusion's multi-step-reasoning limits; verify on your own evals.
Gemini Diffusion (Google)	Closed demo	Impressive demo speeds; no public API as of mid-2026. Do not architect around it.
DiffusionGemma (Google)	Open weights	An open-weight Gemma-family diffusion model, reported over 1,000 tokens/sec on one datacenter GPU. Do not conflate it with Gemini Diffusion.
LLaDA / LLaDA 2.0	Open weights	The foundational open model and a later MoE successor; competitive with similar-size AR models, not frontier.
Dream 7B	Open weights	Reported to match or exceed similar-size AR models, with a tunable steps knob and strong planning (Sudoku, Countdown); a fully-open code variant exists.

Hybrids: Draft with Diffusion, Verify with AR

The most production-relevant work fuses the two paradigms:

Diffusion as a speculative drafter. A diffusion model proposes a block of tokens in parallel and an AR model verifies them, preserving the AR target distribution, analogous to speculative decoding. TiDAR (arXiv:2511.08923) reports roughly 5x the tokens per second of AR while closing the quality gap, positioned to beat standard speculative decoding for latency-critical use.
Block / semi-AR adaptation. Fast-dLLM v2 (arXiv:2509.26328) adapts an existing AR model (Qwen) into a block-diffusion model with a block-wise KV cache and confidence-aware parallel decoding, the practical recipe for getting cacheable, fast diffusion decoding out of checkpoints you already have.
RL for diffusion reasoning. d1 (arXiv:2504.12216) is the first GRPO-style RL for masked diffusion LLMs (diffu-GRPO), pairing masked SFT with an efficient RL stage, and reports consistent gains over the base diffusion model on math and planning benchmarks.

The likely near-term production path is not pure diffusion but these hybrids, which recover KV caching and quality while keeping much of the speed.

Where Diffusion Wins and Where AR Still Wins

Diffusion advantages (structural, not just hype):

Low-latency, high-throughput serving, the headline use case, where round-trip latency dominates (interactive coding, autocomplete, real-time agents).
Code, where generation is iterative back-and-forth refinement rather than strictly sequential.
Infilling and multi-region editing, native to the masking objective: mask the relevant locations and regenerate all of them at once (for example, add a parameter consistently across a signature, its docstring, call sites, and tests in one shot) instead of orchestrating multiple AR calls.
Reversal and order-insensitive reasoning and planning, where bidirectional context helps.

Where AR still wins in 2026: long context and agentic/tool-use workflows (a 2026 reality-check study documents diffusion weaknesses in long-context handling, KV-cache efficiency, and function calling); frontier quality on knowledge and hard reasoning; and very long or variable-length generation, which pure diffusion handles poorly and block/semi-AR variants exist to patch. AR also has years of serving-stack and KV-cache optimization that diffusion's ecosystem has not yet matched.

Maturity and What to Do Today

The honest read: mostly early, with a thin layer of real commercial offerings on a fast-moving open research base. What an engineer should do:

Do not replace your AR frontier model for quality-critical, long-context, or agentic work.
Do pilot diffusion where latency and throughput dominate and outputs are short-to-medium and structured, code completion and infill, bulk transforms, structured extraction. A commercial OpenAI-compatible diffusion API is the lowest-friction A/B test; an open-weight model if you need to self-host.
Watch the hybrids, not just pure diffusion, since block/semi-AR plus diffusion-drafting-with-AR-verification is the likely production path.
Benchmark on your tasks with realistic decoding settings. The quality you get is a direct function of how aggressively you unmask, and vendor tokens-per-second numbers assume aggressive settings.
Treat "reasoning diffusion model" marketing skeptically and verify on your own reasoning evals.

Interview Questions

Q: How does a diffusion language model differ from an autoregressive one, and where is that an advantage?

Strong answer: An autoregressive model generates left to right, one token per forward pass, each conditioned only on the past. A diffusion language model starts from a fully masked sequence and iteratively denoises all positions in parallel over a few steps, with bidirectional attention. The practical advantage is throughput: it commits many tokens per forward pass instead of one, so reported speeds run over 1,000 tokens per second, several times faster than speed-optimized AR. Bidirectionality also helps on infilling and editing, where you mask several locations and regenerate them coherently in one shot, and on order-insensitive tasks. The cost is a quality gap on knowledge and hard reasoning, partly because masked diffusion optimizes a likelihood bound rather than exact likelihood, and weaker long-context and tool-use behavior. So I would reach for it on latency-critical, short-to-medium, structured generation like code, not as a frontier-model replacement.

Q: If diffusion LLMs are faster, why has not everyone switched?

Strong answer: Because the speed comes with real tradeoffs and the ecosystem is young. The quality gap is still meaningful on knowledge and hard reasoning, and the speed itself is a function of how aggressively you unmask: the best quality comes from committing fewer tokens per step, which gives back much of the speed. Pure diffusion also handles long context and variable-length output poorly and lacks the mature KV-cache and serving optimizations AR has accumulated. The likely path is not a wholesale switch but hybrids, block or semi-autoregressive models and diffusion-as-a-speculative-drafter, that recover caching and quality while keeping much of the throughput. In 2026 the pragmatic move is to pilot diffusion where latency dominates and outputs are structured, and keep the AR frontier model for everything else.

References

Nie et al., "Large Language Diffusion Models" (LLaDA) arXiv:2502.09992
Arriola et al., "Block Diffusion" (BD3-LMs) arXiv:2503.09573
"TiDAR: Think in Diffusion, Talk in Autoregression" arXiv:2511.08923
"Fast-dLLM v2" arXiv:2509.26328
Zhao et al., "d1: Scaling Reasoning in Diffusion LLMs via RL" arXiv:2504.12216
"Dream 7B" arXiv:2508.15487
Inception Labs, Mercury (commercial diffusion LLM)

Next: On-Device and Edge Deployment