Continuous Latent Diffusion Language Model
Hongcan Guo, Qinyu Zhao, Yian Zhao, Shen Nie, Rui Zhu, Qiushan Guo, Feng Wang, Tao Yang, Hengshuang Zhao, Guoqiang Wei, Yan Zeng
arXiv:2605.06548 · 85▲ · cs.CL, cs.AI, cs.CV
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
CoLa DLM is a text generation model that replaces left-to-right autoregression with a three-part pipeline: a Text VAE maps text to continuous latent variables, a block-causal Diffusion Transformer (DiT) learns a semantic prior over those latents via Flow Matching, and a conditional decoder turns generated latents back into text. The key design choice is that diffusion transports a latent prior (global semantics) rather than recovering token-level observations, separating semantic planning from token realization.
Why it matters
This is mostly a research-direction paper rather than a deployment-ready tool. It argues that non-autoregressive text generation can scale, using block-causal attention (bidirectional within a block, causal across blocks) to allow parallel computation inside each block instead of strictly sequential token decoding, and it demonstrates the latent-prior approach naturally extends to continuous modalities like images.
Practical takeaway
Watch this as an alternative to autoregressive and discrete diffusion (LLaDA-style) language models, but note it is still at the ~2B-parameter research stage with no released production tooling. One useful practical observation from the paper: perplexity/likelihood correlates poorly with actual generation quality here, so the authors evaluate with few-shot accuracy instead.
Key result
The authors trained CoLa DLM against strictly matched ~2B-parameter autoregressive (LLaMA-style) and LLaDA baselines with identical data, tokenizer, and optimization, and report favorable scaling curves up to about 2000 EFLOPs across 8 benchmarks (LAMBADA, MMLU, SIQA, SQuAD, HellaSwag, etc.). The paper claims strong scaling behavior rather than a single clean win number, and comparisons are limited to models the authors trained themselves at ~2B scale.
Jargon
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.