Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers
Pengqi Lu
arXiv:2605.06169 · 238▲ · cs.LG, cs.CV
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
The paper diagnoses a training collapse in very deep Diffusion Transformers where token representations homogenize toward their sequence mean, which the author calls Mean Mode Screaming, and traces it to a gradient decomposition where a mean-coherent component grows as O(T) once tokens align. It proposes MV-Split Residuals, a modified Post-Norm residual merge that applies separate learnable gains to the centered (token-varying) part of the residual branch versus a leaky replacement of the trunk mean.
Why it matters
If you are trying to scale DiT depth into the hundreds or thousands of layers, standard residual stabilizers like LayerScale keep training from diverging but also damp the useful token-varying signal, slowing convergence. This mostly matters to researchers and teams doing ultra-deep generative model training; the fix is a drop-in change to the residual connection rather than a new optimizer or normalization scheme.
Practical takeaway
If you see a deep Post-Norm Transformer train stably for thousands of steps then suddenly diverge back to initialization loss without NaNs, watch for mean-dominated token collapse (rising cosine similarity across tokens, Q/K gradients dropping several orders of magnitude). You can try splitting the residual gain so the token-mean path is damped separately from the centered path instead of shrinking the whole branch.
Key result
On a matched 400-layer DiT on ImageNet 256x256, MV-Split reaches FID 2.60 / IS 185.5 at 50k steps versus LayerScale's 2.90 / 165.5, while the un-stabilized baseline diverges before the first checkpoint. The 1000-layer result (FID 2.77) is presented only as a scale-validation run on a separate training pipeline, not a matched comparison, so it is not directly comparable to the 400-layer numbers.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.