Learning from the Self-future: On-policy Self-distillation for dLLMs
Yifu Luo, Zeyu Chen, Haoyu Wang, Xinhao Hu, Yuxuan Zhang, Zhizhou Sha, Shiwei Liu
arXiv:2606.18195 · 76▲ · cs.CL
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
The paper introduces d-OPSD, an on-policy self-distillation method for diffusion language models (dLLMs) where a single model acts as both student and teacher. The teacher gets privileged information by revealing part of the model's own already-generated answer as a suffix condition, and training minimizes step-level KL divergence between student and teacher across the iterative denoising steps instead of the token-level divergence used for autoregressive models.
Why it matters
For anyone post-training diffusion LLMs, this offers a path to better reasoning without the heavy compute of reinforcement learning. The paper reports reaching comparable or better accuracy while using roughly 10% of the optimization steps that a GRPO-style RLVR baseline needs, which cuts training cost.
Practical takeaway
If you are fine-tuning a masked diffusion LLM like LLaDA, you can try using the model's own correct completions as suffix conditioning to build a self-teacher rather than appending static reference solutions, and supervise per denoising step. Watch for policy collapse after peak performance, which the authors report as a failure mode shared with RLVR.
Key result
On GSM8K with LLaDA-8B-Instruct, d-OPSD reaches 81.0 (256 seq len) versus 79.8 for the diffu-GRPO RLVR baseline and 76.0 for the base model, while converging in about 425 optimization steps versus 7700 for diffu-GRPO. Caveat: gains are modest and evaluations cover only four reasoning tasks on a single 8B model, with best-checkpoint selection reported.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.