Paper
Auto-Encoding Variational Bayes
Diederik P Kingma, Max Welling
arXiv:1312.6114 · 0▲ · stat.ML, cs.LG
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
This paper introduces the reparameterization trick (rewriting a latent variable z as a deterministic function of the parameters plus fixed noise) so that the variational lower bound becomes differentiable and trainable with ordinary stochastic gradient descent. Applying this to an encoder/decoder pair of neural networks gives the variational auto-encoder (VAE), trained end to end with the AEVB algorithm.
Why it matters
Before this, fitting directed latent-variable models with intractable posteriors required slow per-datapoint MCMC or restrictive mean-field approximations. The reparameterization trick lets you learn a generative model and an amortized inference network jointly with minibatch SGD, so inference on new data is a single forward pass instead of an iterative sampling loop.
Practical takeaway
This is the foundational technique behind VAEs and much of modern deep generative modeling and latent-variable inference; if you want continuous latent representations you can train an encoder that produces a mean and variance, sample via z = mu + sigma * epsilon, and backprop through it. The reparameterization trick still shows up anywhere you need gradients through a sampling step.
Key result
On MNIST and Frey Face, AEVB reached a higher variational lower bound and converged faster than the wake-sleep algorithm across latent dimensionalities from 2 to 200, and matched or beat Monte Carlo EM on estimated marginal likelihood. The experiments are small by current standards (two image datasets, single-hidden-layer MLPs, marginal likelihood estimates only reliable below 5 latent dimensions).
Jargon
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.