An Overview of Diffusion Language Models

Discrete diffusion language models learn to restore corrupted token sequences; update order and iteration count affect quality and cost.

A different generation order

Autoregressive (AR) models generate each token conditioned on preceding tokens. A KV cache avoids recomputing earlier keys and values. It does not make dense-attention reads of the growing cache constant in cost; a typical generation loop appends tokens after the committed prefix.

Diffusion language models learn to restore data from corrupted sequences. Corruption can replace tokens with other tokens or masks. Not every diffusion model uses the same masking or revision procedure.

Established foundations of discrete diffusion

Austin et al.'s D3PM, published at NeurIPS 2021, studies structured corruption transitions in discrete spaces. It includes transitions toward an absorbing state as well as uniform replacement. Using a mask as the absorbing state is one option.

Lou et al.'s SEDD, published at ICML 2024, proposes score entropy for learning ratios of discrete probabilities. Sahoo et al.'s MDLM, published at NeurIPS 2024, simplifies masked diffusion training into a mixture of masked-language-modeling losses. These methods use different objectives and samplers.

Their language-modeling evaluations establish concrete training and generation methods. They do not establish superiority on every reasoning task or commercial serving speed. Comparing unrelated models' maximum throughput is also not a validation of the theory.

Generation with a masked model

One form of masked diffusion starts from masks and gradually fills multiple positions. The model computes token distributions from the currently observed state. The sampler determines when and where positions are committed.

flowchart LR
 A["Masked sequence"] --> B["Compute distributions from current state"]
 B --> C["Update selected positions using the sampler"]
 C --> D{"Masks remain?"}
 D -->|Yes| B
 D -->|No| E["Completed sequence"]

Distinguish samplers that retain committed tokens from variants that remask them. Repeated model calls do not establish unrestricted revision of every token. Infilling also requires a compatible conditional-generation procedure and sampler.

MDLM includes semi-autoregressive sampling for variable-length generation. This establishes that block-based generation is possible. It does not establish near-universal adoption of one method in production.

Conditions for speed and memory comparisons

The number of function evaluations (NFE) counts neural-network calls. Fewer calls can reduce computation, but sequence length per call, kernels, and batching also affect runtime. NFE alone cannot determine speedup over an AR model.

Applying full dense attention to length (L) for (T) steps has a length-and-step computation term of (O(TL^2)). This holds model dimensions fixed. Reusable inference buffers mean peak memory does not necessarily increase by a factor of (T).

Evaluate complete execution at matched target quality to establish speed. Reported results with different architectures, samplers, and hardware cannot isolate one optimization's causal effect. This article does not reproduce the papers' performance.

Summary

D3PM, SEDD, and MDLM provide established formulations of discrete corruption, objectives, and sampling. Diffusion generation can update multiple positions, but revision and cache-reuse conditions differ by method. Parallel prediction and fewer network calls do not guarantee higher throughput. Compare quality, latency, throughput, and peak memory under matched conditions.