Theory

Diederik P. Kingma, Jimmy Ba

This paper introduces Adam, a first-order stochastic optimization algorithm that maintains per-parameter adaptive learning rates using exponential moving averages of the gradient (first moment) and squared gradient (second raw moment), with a bias-correction step for the zero-initialized averages. It combines ideas from AdaGrad (sparse gradients) and RMSProp (non-stationary objectives), and also presents AdaMax, a variant based on the infinity norm.

Diederik P Kingma, Max Welling

This paper introduces the reparameterization trick (rewriting a latent variable z as a deterministic function of the parameters plus fixed noise) so that the variational lower bound becomes differentiable and trainable with ordinary stochastic gradient descent. Applying this to an encoder/decoder pair of neural networks gives the variational auto-encoder (VAE), trained end to end with the AEVB algorithm.

Hongcan Guo, Qinyu Zhao, Yian Zhao, Shen Nie, Rui Zhu, Qiushan Guo, Feng Wang, Tao Yang, Hengshuang Zhao, Guoqiang Wei, Yan Zeng

CoLa DLM is a text generation model that replaces left-to-right autoregression with a three-part pipeline: a Text VAE maps text to continuous latent variables, a block-causal Diffusion Transformer (DiT) learns a semantic prior over those latents via Flow Matching, and a conditional decoder turns generated latents back into text. The key design choice is that diffusion transports a latent prior (global semantics) rather than recovering token-level observations, separating semantic planning from token realization.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio

This is the original Generative Adversarial Nets (GAN) paper. It trains two neural networks simultaneously in a minimax game: a generator G that maps random noise to fake samples, and a discriminator D that tries to tell real training data from generated data, with both trained by ordinary backpropagation.

Pengqi Lu

The paper diagnoses a training collapse in very deep Diffusion Transformers where token representations homogenize toward their sequence mean, which the author calls Mean Mode Screaming, and traces it to a gradient decomposition where a mean-coherent component grows as O(T) once tokens align. It proposes MV-Split Residuals, a modified Post-Norm residual merge that applies separate learnable gains to the centered (token-varying) part of the residual branch versus a leaky replacement of the trunk mean.