Training

Diederik P. Kingma, Jimmy Ba

This paper introduces Adam, a first-order stochastic optimization algorithm that maintains per-parameter adaptive learning rates using exponential moving averages of the gradient (first moment) and squared gradient (second raw moment), with a bias-correction step for the zero-initialized averages. It combines ideas from AdaGrad (sparse gradients) and RMSProp (non-stationary objectives), and also presents AdaMax, a variant based on the infinity norm.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby

This paper introduces the Vision Transformer (ViT), which applies a standard Transformer encoder directly to image classification by splitting an image into fixed-size patches (for example 16x16 pixels), linearly embedding each patch as a token, adding position embeddings, and processing the sequence with self-attention. It deliberately avoids convolutions and image-specific inductive biases except for the initial patch extraction and position embedding interpolation at fine-tuning time.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

This paper introduces the Transformer, a sequence-to-sequence architecture for machine translation that replaces recurrent and convolutional layers entirely with stacked self-attention and position-wise feed-forward layers. It uses scaled dot-product attention with multiple attention heads, plus sinusoidal positional encodings to inject word order since there is no recurrence.

Zhifei Xie, Zihang Liu, Ze An, Xiaobin Hu, Yue Liao, Ziyang Ma, Dongchao Yang, Mingbao Lin, Deheng Ye, Shuicheng Yan, Chunyan Miao

The paper builds Audio-Interaction, a streaming audio language model (fine-tuned from Qwen2.5-Omni-3B) that consumes audio in 400ms chunks and predicts a per-chunk control token (<silent> or <response>) to decide whether to speak or keep listening. It is trained via a framework called SoundFlow using a 2.6M-item synthetic streaming corpus (StreamAudio-2M) and served with a FIFO-scheduled asynchronous inference scheme that decouples encoding from decoding.

Diederik P Kingma, Max Welling

This paper introduces the reparameterization trick (rewriting a latent variable z as a deterministic function of the parameters plus fixed noise) so that the variational lower bound becomes differentiable and trainable with ordinary stochastic gradient descent. Applying this to an encoder/decoder pair of neural networks gives the variational auto-encoder (VAE), trained end to end with the AEVB algorithm.

Sergey Ioffe, Christian Szegedy

This paper introduces Batch Normalization, a layer inserted before each nonlinearity that normalizes activations to zero mean and unit variance using per-mini-batch statistics, then applies a learned scale and shift. The normalization is part of the network architecture so gradients backpropagate through it, and at inference time it uses population statistics computed from moving averages.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

BERT is a language representation model that pre-trains a deep bidirectional Transformer encoder using two self-supervised tasks: masked language modeling (predicting randomly hidden tokens from both left and right context) and next sentence prediction. The pre-trained model is then fine-tuned end-to-end for specific tasks by adding a single output layer, with no task-specific architecture changes.

Lennart Purucker, Andrej Tschalzev, Nick Erickson, Gioia Blayer, David Holzmüller, Alan Arazi, Alexander Pfefferle, Mustafa Tajjar, Gaël Varoquaux, Frank Hutter

The paper introduces BeyondArena, a benchmark of 142 manually curated tabular datasets that spans IID, temporal, and grouped (non-IID) prediction tasks across sample sizes from 100 to 1 million rows, plus DataFoundry, a Python framework for reproducible dataset curation. It evaluates 11 models including three tabular foundation models (TabPFN-2.6, TabICLv2, TabDPT) against gradient-boosted trees and MLPs using in-context learning for the foundation models and tuning plus ensembling for the traditional ones.

Min Zhao, Hongzhou Zhu, Kaiwen Zheng, Zihan Zhou, Bokai Yan, Xinyuan Li, Xiao Yang, Chongxuan Li, Jun Zhu

The paper distills bidirectional video diffusion models into few-step autoregressive students for real-time interactive video generation, replacing the expensive causal ODE initialization step of prior work (Causal Forcing) with causal consistency distillation (causal CD). Causal CD gets its training signal from a single online teacher ODE step between adjacent timesteps on real videos, avoiding the need to precompute and store full PF-ODE trajectories.

Wei Zhou, Xiongwei Zhu, Zelin Xu, Bo Dong, Lixue Gong, Yongyuan Liang, Meng Chu, Leigang Qu, Lingdong Kong, Wei Liu, Tat-Seng Chua

DanceOPD is an on-policy distillation method for flow-matching image generators that combines multiple frozen capability models (text-to-image, local editing, global editing) into one student. Each training sample is hard-routed to exactly one teacher's velocity field, that field is queried on a single low-noise state from the student's own rollout (stop-gradient), and the student is trained with plain velocity MSE.

Kaiyi Zhang, Wei Wu, Yankai Lin

The paper reframes RLVR (reinforcement learning from verifiable rewards) policy updates as an implicit linear discriminator over token-gradient vectors, arguing that standard sequence-level methods like DAPO let shared high-frequency tokens (formatting, entities) dominate the update direction. It proposes DelTA, which computes per-token coefficients based on how well each token's gradient distinguishes positive-advantage from negative-advantage responses, then uses those coefficients to reweight the DAPO surrogate loss.

Jonathan Ho, Ajay Jain, Pieter Abbeel

This paper introduces Denoising Diffusion Probabilistic Models (DDPMs), a generative model that learns to reverse a fixed Markov chain that gradually adds Gaussian noise to images. The key practical simplification is training a U-Net to predict the noise added at each timestep using a plain weighted mean-squared-error objective, which the authors show is equivalent to denoising score matching over multiple noise levels.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn

This paper introduces Direct Preference Optimization (DPO), a method for aligning language models to human preferences using a single binary cross-entropy classification loss instead of the standard RLHF pipeline. It shows that the KL-constrained reward maximization objective used in RLHF has a closed-form optimal policy, which lets you reparameterize the reward as a function of the policy itself and skip both the separate reward model and the reinforcement learning loop.

Nan Chen, Yiyang Cai, Rongchang Xie, Junwen Pan, Cheng Chen, Weinan Jia, Zhuowei Chen, Wen Zhou, Zhenbang Sun, Wenhan Luo

DomainShuttle is a subject-driven text-to-video system built on the Wan2.1/2.2 DiT video models that adds three components: Domain-MoT (separate transformer branches for reference images versus video plus a domain-aware AdaLN modulation), Video-Reference DualRoPE (placing reference tokens in a separate positional encoding space from video tokens), and a Cross-Pair Consistent Loss that aligns two different reference sets of the same subject. The goal is to preserve a reference subject's intrinsic features while still allowing style, domain, and semantic changes driven by the text prompt (cross-domain generation), rather than only copying the reference (in-domain).

Guochao Jiang, Jingyi Song, Guofeng Quan, Chuzhan Hao, Guohua Liu, Yuewei Zhang

DVAO is a method for combining multiple reward signals in GRPO-style RL training of LLMs. Instead of using fixed weights to combine rewards or advantages, it scales each objective's contribution by that objective's empirical reward standard deviation within a rollout group, up-weighting objectives that show more variation (a stronger learning signal) and down-weighting noisy or saturated ones.

Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

This is the original word2vec paper. It introduces two log-linear model architectures, Continuous Bag-of-Words (CBOW) and Skip-gram, that learn dense word vectors by removing the expensive non-linear hidden layer used in prior neural language models and training on billions of words.

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré

FlashAttention is an exact attention algorithm that restructures the computation to avoid writing the large N x N attention matrix to GPU high bandwidth memory (HBM). It uses tiling to compute softmax block by block in fast on-chip SRAM and recomputes the attention matrix during the backward pass instead of storing it, all fused into a single CUDA kernel.

Zhen Fang, Wenxuan Huang, Yu Zeng, Yiming Zhao, Shuang Chen, Kaituo Feng, Yunlong Lin, Lin Chen, Zehui Chen, Shaosheng Cao, Feng Zhao

Flow-OPD is a post-training method for text-to-image flow matching models that replaces multi-reward reinforcement learning with on-policy distillation from multiple single-task teacher models. It trains domain-expert teachers via single-reward GRPO, then distills them into one student by routing prompts to the relevant teacher and matching velocity fields, showing that the reverse-KL divergence between student and teacher policies reduces analytically to a weighted MSE loss on the vector fields.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio

This is the original Generative Adversarial Nets (GAN) paper. It trains two neural networks simultaneously in a minimax game: a generator G that maps random noise to fake samples, and a discriminator D that tries to tell real training data from generated data, with both trained by ordinary backpropagation.

Guan Wang, Changling Liu, Chenyu Wang, Cai Zhou, Yuhao Sun, Yifei Wu, Shuai Zhen, Luca Scimeca, Yasin Abbasi Yadkori

HRM-Text replaces the standard decoder-only Transformer with a Hierarchical Recurrent Model that splits computation into a slow strategic (H) module and a fast execution (L) module, stabilized for language via two tricks the authors call MagicNorm (a hybrid Pre/Post normalization scheme) and a warmup schedule for truncated backpropagation through time. Instead of pretraining on raw web text, it trains a 1B-parameter model from scratch only on instruction-response pairs, computing loss just on the response (a task-completion objective) with a PrefixLM attention mask that lets instruction tokens attend bidirectionally.

Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov

This is the original dropout paper: during training, each hidden unit is randomly omitted with probability 0.5 on every training case, which stops units from co-adapting to rely on specific other units. At test time you use the full network with outgoing weights halved, which approximates averaging over the exponentially many thinned networks.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei

This is the GPT-3 paper: the authors train an autoregressive transformer language model with 175 billion parameters and evaluate it on over two dozen NLP tasks using in-context learning, where the model gets task demonstrations in its prompt at inference time with no gradient updates. They test three settings (zero-shot, one-shot, and few-shot with 10 to 100 examples in the context window) across eight model sizes to measure how performance scales.

Yifu Luo, Zeyu Chen, Haoyu Wang, Xinhao Hu, Yuxuan Zhang, Zhizhou Sha, Shiwei Liu

The paper introduces d-OPSD, an on-policy self-distillation method for diffusion language models (dLLMs) where a single model acts as both student and teacher. The teacher gets privileged information by revealing part of the model's own already-generated answer as a suffix condition, and training minimizes step-level KL divergence between student and teacher across the iterative denoising steps instead of the token-level divergence used for autoregressive models.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever

This is the CLIP paper. It trains an image encoder and a text encoder jointly on 400 million (image, text) pairs scraped from the internet, using a contrastive objective that predicts which caption goes with which image within a batch rather than predicting exact caption words. After pre-training, you build a classifier for any dataset by feeding the class names (as text prompts) through the text encoder, so the model classifies images it was never explicitly trained to label.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample

LLaMA is a family of transformer language models (7B to 65B parameters) trained only on publicly available datasets, using architectural changes like pre-normalization with RMSNorm, SwiGLU activations, and rotary positional embeddings. The models are trained on far more tokens than Chinchilla scaling laws recommend (up to 1.4T tokens), trading extra training compute for cheaper inference.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen

LoRA (Low-Rank Adaptation) freezes a pre-trained model's weights and injects trainable rank-decomposition matrices (a pair of small matrices B and A whose product approximates the weight update) into each Transformer layer, so only those small matrices are trained. In their Transformer experiments the authors apply it mostly to the query and value attention projection matrices.

Pengqi Lu

The paper diagnoses a training collapse in very deep Diffusion Transformers where token representations homogenize toward their sequence mean, which the author calls Mean Mode Screaming, and traces it to a gradient decomposition where a mean-coherent component grows as O(T) once tokens align. It proposes MV-Split Residuals, a modified Post-Norm residual merge that applies separate learnable gains to the centered (token-varying) part of the residual branch versus a leaky replacement of the trunk mean.

Zhifei Xie, Kaiyu Pang, Haobin Zhang, Deheng Ye, Xiaobin Hu, Shuicheng Yan, Chunyan Miao

Mega-ASR fine-tunes Qwen3-ASR-1.7B to handle heavily degraded real-world audio using two techniques: Acoustic-to-Semantic Progressive Supervised Fine-Tuning (a WER-graded curriculum that trains the encoder, then the LLM, then jointly) and Dual-Granularity WER-Gated Policy Optimization (a reinforcement learning reward that switches between token-level and sentence-level scoring based on WER). It is trained on Voices-in-the-Wild-2M, a 2.4M-clip synthetic dataset built by simulating 7 atomic acoustic effects and 54 compound scenarios at the spectrogram level.

Alan Arazi, Eilam Shapira, Shoham Grunblat, Mor Ventura, Elad Hoffer, Gioia Blayer, David Holzmüller, Lennart Purucker, Gaël Varoquaux, Frank Hutter, Roi Reichart

MulTaBench is a benchmark of 40 datasets (20 image-tabular, 20 text-tabular) for multimodal tabular learning, curated by an automated pipeline that only keeps datasets where each modality adds independent predictive signal and where task-specific tuning of the encoder beats frozen off-the-shelf embeddings. The curation compares four conditions (unimodal, joint frozen, joint target-aware) across five tabular learners, using LoRA finetuning of the top 3 layers of e5 (text) and DINO-v3 (image) encoders as a preprocessing step to produce Target-Aware Representations.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean

The paper introduces the Sparsely-Gated Mixture-of-Experts (MoE) layer, a neural network component containing up to thousands of feed-forward expert sub-networks where a trainable gating network selects only a few experts to run per input token. It uses noisy top-k gating plus auxiliary load-balancing losses, and applies the layer between stacked LSTM layers for language modeling and machine translation.

Lei Bai, Zongsheng Cao, Yang Chen, Zhiyao Cui, Shangheng Du, Yue Fan, Shiyang Feng, Zijie Guo, Haonan He, Liang He, Xiaohan He, Shuyue Hu, Yusong Hu, Songtao Huang, Yichen Jiang, Hao Li, Xin Li, Dahua Lin, Weihao Lin, Fenghua Ling, Dongrui Liu, Zhuo Liu, Runmin Ma, Chunjiang Mu, Haoyang Peng, Tianshuo Peng, Jinxin Shi, Luohe Shi, Boyuan Sun, Zelin Tan, Shengji Tang, Qianyi Wang, Yiming Wu, Yi Xie, Xiangchao Yan, Jingqi Ye, Peng Ye, Fangchen Yu, Jiakang Yuan, Bihao Zhan, Bo Zhang, Chen Zhang, Shufei Zhang, Shuaiyu Zhang, Wenlong Zhang, Yiqun Zhang, Junpeng Zhao, Zhijie Zhong, Bowen Zhou, Yuhao Zhou

A recipe for training a 35B mixture-of-experts agent (Agents-A1) to handle long-horizon agentic tasks by scaling trajectory length rather than parameters. The method has three stages: full-domain supervised fine-tuning on ~100K long trajectories (average 45K tokens), training domain-specific teacher models, then consolidating them into one student via domain-routed on-policy distillation with salient vocabulary alignment (SVA), which aligns student and teacher on the teacher's top-k token support instead of just the sampled token.

Haiwen Diao, Penghao Wu, Hanming Deng, Jiahao Wang, Shihao Bai, Silei Wu, Weichen Fan, Wenjie Ye, Wenwen Tong, Xiangyu Fan, Yan Li, Yubo Wang, Zhijie Cao, Zhiqian Lin, Zhitao Yang, Zhongang Cai, Yuwei Niu, Yue Zhu, Bo Liu, Chengguang Lv, Haojia Yu, Haozhe Xie, Hongli Wang, Jianan Fan, Jiaqi Li, Jiefan Lu, Jingcheng Ni, Junxiang Xu, Kaihuan Liang, Lianqiang Shi, Linjun Dai, Linyan Wang, Oscar Qian, Peng Gao, Pengfei Liu, Qingping Sun, Rui Shen, Ruisi Wang, Shengnan Ma, Shuang Yang, Siyi Xie, Siying Li, Tianbo Zhong, Xiangli Kong, Xuanke Shi, Yang Gao, Yongqiang Yao, Yves Wang, Zhengqi Bai, Zhengyu Lin, Zixin Yin, Wenxiu Sun, Ruihao Gong, Quan Wang, Lewei Lu, Lei Yang, Ziwei Liu, Dahua Lin

SenseNova-U1 is a multimodal model that handles both understanding (image/text comprehension) and image generation in a single architecture that operates directly on raw pixels and text, dropping the usual pretrained vision encoder and VAE. It uses a Mixture-of-Transformers (MoT) backbone where a shared attention path processes both streams, training understanding with next-token prediction and generation with pixel-space flow matching. Two variants ship: a dense 8B and a 30B mixture-of-experts model with about 3B active parameters.

Ilya Sutskever, Oriol Vinyals, Quoc V. Le

This paper introduces the sequence-to-sequence (seq2seq) encoder-decoder architecture: one multilayer LSTM reads an input sentence into a fixed-length vector, and a second LSTM decodes that vector into the output sentence one word at a time. It applies this to English-to-French machine translation and adds a key trick of reversing the word order of the source sentence before feeding it in.

Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Yuqing Yang, Dongdong Chen, Xue Yang, Chong Luo

SkillOpt is a text-space optimizer that treats an agent's natural-language skill document as a trainable object, borrowing deep-learning controls like batches, a textual learning rate (bounded number of edits per step), a held-out validation gate, and epoch-wise slow/meta updates. A frontier optimizer model reads execution trajectories, proposes structured add/delete/replace edits to the skill, and only accepts edits that improve validation performance, exporting a compact best_skill.md (300 to 2,000 tokens) while the target model and harness stay frozen.

Bin Wu, Mengqi Huang, Shaojin Wu, Weinan Jia, Yuxin Wang, Zhendong Mao, Yongdong Zhang

Stream-R1 modifies distribution matching distillation (DMD) for streaming autoregressive video generation by reweighting the distillation loss with a pretrained video reward model at two levels: a per-rollout scalar weight (exponential of the reward score) and a per-pixel spatiotemporal weight derived from backpropagating the reward model's gradient. It replaces the uniform weighting that treats every rollout, frame, and pixel as equally reliable supervision with reward-guided weighting that concentrates optimization on regions where refinement helps most.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre

The paper studies how to split a fixed training compute budget between model size and number of training tokens, fitting three empirical estimators (minimum-over-training-curves, IsoFLOP profiles, and a parametric loss function) across over 400 models trained from 70M to 16B parameters. It finds model size and training tokens should scale in roughly equal proportion (doubling one means doubling the other), then validates this by training Chinchilla, a 70B model on 1.4T tokens, at the same compute as the 280B Gopher.

Hanyu Guo, Jiedong Yang, Chao Chen, Longfei Xu, Kaikui Liu, Xiangxiang Chu

TransitLM is a dataset of over 13 million transit route planning records (from Amap logs across four Chinese cities, 120,845 stations) plus a three-task benchmark, released as a continual pre-training corpus and supervised fine-tuning data. The authors validate it by continually pre-training then fine-tuning Qwen3 models (0.6B to 4B) that generate complete transit routes end-to-end from origin-destination GPS coordinates, with each station ID registered as a dedicated vocabulary token so routes cannot be hallucinated via character composition.