An Overview of Diffusion Language Models

Diffusion language models generate text by iteratively refining a noisy sequence rather than predicting tokens left to right.

What autoregression leaves on the table

Autoregressive (AR) language models factor sequence probability from left to right. Each token is predicted conditioned only on preceding tokens, which keeps training simple and gives an exactly defined likelihood. The practical advantage is large too: a key-value cache (KV cache) keeps per-token inference cost nearly constant.

That factorization carries structural constraints with it. Once a token is chosen it cannot be undone, and attention reaches only the left context. Right-to-left inference is weak without separate training, and serial decoding puts a floor under latency.

The best-known symptom is the reversal curse: a model trained on "A is B" answers "who is B" poorly, which follows from the asymmetry of left-to-right causal factorization. Infilling at arbitrary positions and self-correction during generation are also tasks AR does not handle naturally.

The different factorization diffusion proposes

Diffusion models the same joint distribution through iterative refinement. It starts from a sequence corrupted by noise and revisits the same positions repeatedly, correcting them globally. This global refinement capability is why diffusion became standard for images and audio. Diffusion language models (DLMs) carry the same motivation over to text.

The key difference is that decisions stay provisional. AR commits to a token once it is chosen, while diffusion can change the value at a position in a later step. A bidirectional transformer edits while seeing both left and right context, which yields structural advantages in reversal, infilling, and global consistency.

Text is made of discrete tokens, though, so continuous image diffusion does not transfer directly. Continuous-embedding diffusion from 2022 to 2023 ran aground at scale because rounding created a train-inference distribution mismatch and clusters collapsed as vocabularies grew. That failure shifted the center of gravity toward defining diffusion directly in token space.

Diffusion over discrete tokens

D3PM (Discrete Denoising Diffusion Probabilistic Models) is the first general framework to define diffusion in discrete token space, proposed by Austin et al. at NeurIPS 2021. It uses K = |V|+1 states, the vocabulary plus a mask token, and writes the forward transition at each time step as a categorical distribution. Accumulating the transition matrices progressively corrupts a clean sequence, which is the forward noising process.

The choice of transition matrix decides the outcome. A uniform transition spreads noise across all tokens, while an absorbing transition pulls every position toward the mask. The absorbing route is consistently better on perplexity (PPL) and downstream tasks, and the intuition lies in the simplicity of denoising each position exactly once.

The theory simplified into a single line of work after D3PM. SEDD, based on score entropy over a continuous-time Markov chain (CTMC), was the first to close the gap with GPT-2. MDLM then narrowed the loss to masking only, reducing the objective in the absorbing case to a time-weighted negative log likelihood over masked positions. That this loss is formally identical to masked language modeling (MLM) produced a unified picture.

The inference procedure that unmasks

Inference in a diffusion LM starts from a fully masked sequence. At each step the model predicts token distributions for all masked positions at once, commits some of them (unmasking), and leaves the rest masked. The basic skeleton is a predict-unmask-remask cycle that fills in the sequence as time decreases.

The pseudocode below transcribes the same skeleton. As time t decreases, the commit threshold loosens and more positions are released.

# Pseudocode: basic decoding loop of a masked diffusion LM
x = [MASK] * L                       # start fully masked
for t in range(T, 0, -1):            # decrease from time T to 0
    logits = model(x)                # bidirectional attention, all positions predicted at once
    probs = softmax(logits)
    for i in masked_positions(x):
        if confidence(probs[i]) >= threshold(t):
            x[i] = sample(probs[i])  # unmask only high-confidence positions
return x

With identical weights, the decoding policy changes quality and throughput substantially. Top-k confidence, which releases the most confident positions first, is the default, and remasking low-confidence positions adds a self-correction effect. Entropy can also be used to vary the number of function evaluations (NFE) per position. This adaptive approach is used in throughput-oriented workloads.

Production systems have converged almost universally on block-wise decoding. Splitting the sequence into blocks and processing left to right across blocks and by diffusion within blocks gives KV cache compatibility and variable length. Adding self-speculative decoding and consistency distillation on top reduces effective NFE substantially. This step-count optimization is the direct cause of roughly 5x throughput in commercial systems (per the Mercury announcement).

Real models

The masked diffusion model LLaDA (Large Language Diffusion with mAsking) is the first quantitative demonstration at 8B scale. Trained on 2.3 trillion tokens, it matched LLaMA3 8B on math, code, and general tasks (LLaDA paper, arXiv 2502.09992). In the reversal completion evaluation in the same paper, forward and backward accuracy stayed even. Long-form reasoning, where pure diffusion is weak, is compensated for with block decoding.

Commercial and big tech entries pushed the throughput race. Mercury, the first commercial diffusion LLM, reported 1109 tok/s on an H100. Gemini Diffusion then announced 1479 tok/s and Seed Diffusion 2146 tok/s. The numbers come from each organization's own announcements, and hardware and configuration differ, so direct comparison requires caution.

Strengths and limits

The strengths cluster where bidirectional attention is inherently favorable: reversal evaluations, infilling, global consistency, and self-correction. Unlike AR, which requires separate fill-in-the-middle training, infilling fills arbitrary positions in one pass with no extra training.

The weaknesses cluster where causal sequential dependence is essential: long chain-of-thought (CoT) reasoning, variable-length generation, and long context. Some likelihood gap remains, but it does not explain all of the downstream performance difference. The conclusion leans toward likelihood not being a sufficient indicator of capability.

The disadvantage on long context comes from the cost structure. A diffusion LM needs full-sequence attention at every step, costing O(L²·T) for one generation. That is T times as expensive as AR, which handles it in O(L²) with a KV cache. Combining block diffusion with sparse or linear attention is a candidate, but as of 2026 it is still at the demonstration stage.

Summary

Diffusion language models replace left-to-right prediction with iterative refinement, gaining a different set of capabilities in bidirectional attention and parallel decoding. Discrete diffusion theory that began with D3PM simplified into MDLM's time-weighted masked loss, laying the foundation for training at 8B scale. LLaDA showed parity with AR, and Mercury, Gemini Diffusion, and Seed Diffusion demonstrated commercial throughput. Diffusion LMs have become a competing paradigm to AR.

The open question is the nature of the limits. The advantages in reversal, infilling, and throughput are structural. Whether the disadvantages in long-form reasoning and long context are inherent or temporary remains unresolved. Production systems have converged almost universally on block diffusion. That reads as a signal that interpolating between pure diffusion and pure AR is the practical answer for now.