RL

Yafu Li, Runzhe Zhan, Haoran Zhang, Shunkai Zhang, Yizhuo Li, Zhilin Wang, Jiacheng Chen, Futing Wang, Xuyang Hu, Yuchen Fan, Bangjie Xu, Yucheng Su, Xinmiao Han, Chenxi Li, Haodi Lei, Yufeng Zhao, Zejin Lin, Qianjia Cheng, Tong Zhu, Xiaoye Qu, Ganqu Cui, Peng Ye, Yun Luo, Zhouchen Lin, Yu Qiao, Bowen Zhou, Ning Ding, Yu Cheng

The paper describes a training recipe that turns a 30B-A3B post-trained reasoning model into an olympiad-level math and physics solver, using three stages: supervised fine-tuning ordered by reverse-perplexity curriculum, a two-stage RL pipeline (verifiable-reward RL followed by proof-quality RL with a generative reward model, self-refinement, and experience replay), and a test-time solve-verify-refine loop. The resulting model, SU-01, is trained on 338K sub-8K-token SFT trajectories plus 200 RL steps.

Kaiyi Zhang, Wei Wu, Yankai Lin

The paper reframes RLVR (reinforcement learning from verifiable rewards) policy updates as an implicit linear discriminator over token-gradient vectors, arguing that standard sequence-level methods like DAPO let shared high-frequency tokens (formatting, entities) dominate the update direction. It proposes DelTA, which computes per-token coefficients based on how well each token's gradient distinguishes positive-advantage from negative-advantage responses, then uses those coefficients to reweight the DAPO surrogate loss.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn

This paper introduces Direct Preference Optimization (DPO), a method for aligning language models to human preferences using a single binary cross-entropy classification loss instead of the standard RLHF pipeline. It shows that the KL-constrained reward maximization objective used in RLHF has a closed-form optimal policy, which lets you reparameterize the reward as a function of the policy itself and skip both the separate reward model and the reinforcement learning loop.

Guochao Jiang, Jingyi Song, Guofeng Quan, Chuzhan Hao, Guohua Liu, Yuewei Zhang

DVAO is a method for combining multiple reward signals in GRPO-style RL training of LLMs. Instead of using fixed weights to combine rewards or advantages, it scales each objective's contribution by that objective's empirical reward standard deviation within a rollout group, up-weighting objectives that show more variation (a stronger learning signal) and down-weighting noisy or saturated ones.

Yifu Yuan, Yaoting Huang, Xianze Yao, Yutong Li, Shuoheng Zhang, Linqi Han, Pengyi Li, Jiangeng Sun, Wenting Jia, Zhao Zhang, Yuhao Liu, Ruihao Liao, Yucheng Hu, Qiyu Wu, Yuxiao Li, Zibin Dong, Fei Ni, Yan Zheng, Shuyang Gu, Yi Ma, Hongyao Tang, Han Hu, Jianye Hao

Embodied-R1.5 is an 8B-parameter vision-language model fine-tuned to unify three robotics-relevant capabilities (spatial reasoning, task planning/correction, and pointing/trajectory generation) in one model, trained via supervised fine-tuning followed by reinforcement learning with verifiable rewards on 15B tokens of mostly auto-generated data. It also ships a Planner-Grounder-Corrector loop where a single model instance plays all three roles for long-horizon tasks, and can be extended into a robot action policy (VLA) by attaching a flow-matching action head.

Zhilin Wang, Han Song, Runzhe Zhan, Jusen Du, Jiacheng Chen, Tianle Li, Qingyu Yin, Yulun Wu, Zhennan Shen, Tong Zhu, Yanshu Li, Guanjie Chen, Derek F. Wong, Yafu Li, Yu Cheng, Yang Yang

EvoPolicyGym is a benchmark that measures how well a coding agent (a language model plus its tool-using harness) can iteratively rewrite an executable decision policy for a reinforcement learning environment, using feedback from sandboxed rollouts under a fixed budget of 128 episodes. Agents edit a Python policy that exposes reset and act(obs), submit it for scored train rollouts, and get judged on held-out return of the checkpoint the server picks via hidden validation.

Zhen Fang, Wenxuan Huang, Yu Zeng, Yiming Zhao, Shuang Chen, Kaituo Feng, Yunlong Lin, Lin Chen, Zehui Chen, Shaosheng Cao, Feng Zhao

Flow-OPD is a post-training method for text-to-image flow matching models that replaces multi-reward reinforcement learning with on-policy distillation from multiple single-task teacher models. It trains domain-expert teachers via single-reward GRPO, then distills them into one student by routing prompts to the relevant teacher and matching velocity fields, showing that the reverse-KL divergence between student and teacher policies reduces analytically to a weighted MSE loss on the vector fields.

Alireza Salemi, Chang Zeng, Atharva Nijasure, Jui-Hui Chung, Razieh Rahimi, Fernando Diaz, Hamed Zamani

GrepSeek trains a compact LLM (Qwen3.5-9B) to answer questions by issuing Unix shell commands (rg, grep, head, awk, etc.) directly against a raw text corpus instead of querying a precomputed retrieval index. It uses a two-stage pipeline: supervised fine-tuning on a synthetic cold-start dataset built by an answer-aware Tutor and answer-blind Planner, followed by GRPO reinforcement learning, plus a sharded-parallel execution engine that runs shell pipelines across corpus shards while preserving byte-exact output.

Pengcheng Jiang, Zhiyi Shi, Kelly Hong, Xueqiang Xu, Jiashuo Sun, Jimeng Sun, Hammad Bashir, Jiawei Han

Harness-1 is a 20B search agent (built on gpt-oss-20b) trained with reinforcement learning inside a stateful harness that moves search bookkeeping out of the model and into the environment. The environment maintains a candidate pool, an importance-tagged curated set, an evidence graph, verification records, and deduplicated/compressed observations, while the policy only decides what to search, which documents to keep, what to verify, and when to stop.

Yifu Luo, Zeyu Chen, Haoyu Wang, Xinhao Hu, Yuxuan Zhang, Zhizhou Sha, Shiwei Liu

The paper introduces d-OPSD, an on-policy self-distillation method for diffusion language models (dLLMs) where a single model acts as both student and teacher. The teacher gets privileged information by revealing part of the model's own already-generated answer as a suffix condition, and training minimizes step-level KL divergence between student and teacher across the iterative denoising steps instead of the token-level divergence used for autoregressive models.

Hongyuan Adam Lu, Z. L. Victor Wei, Qun Zhang, Jinrui Zeng, Bowen Cao, Lingwei Meng, Mocheng Li, Zezhong Wang, Haonan Yin, Naifu Xue, Minyu Chen, Cenyuan Zhang, Zefan Zhang, Hao Wei, Jiawei Zhou, Haoran Xu, Hao Yang, Ronglai Zuo, Tongda Xu, Yonghao Li, Jian Chen, Hebin Wang, Zeyu Gao, Yang Li, Wei Zhao, Qimin Zhong, Siqi Liu, Yumeng Zhang, Leyan Cui, Zhangyu Wang, Wai Lam

The paper applies looped transformers (a shared transformer block iterated repeatedly across depth) to world models, calling the result LoopWM. It adds a spectrally constrained state-retention matrix to keep latent updates bounded, Poisson-sampled variable loop depth during training, an early-exit gate for adaptive inference, and a deferred-decoding mode that skips intermediate observation reconstruction during multi-step rollouts.

Jiacheng Chen, Xinyu Zhang, Shunkai Zhang, Yanmohan Wang, Lin Li, Tiancheng Qin, Qin Wang, Zhengmao Zhu, Tianle Li, Jingyang Li, Zehan Li, Binyang Jiang, Jin Zhu, Han Ding, Fei Yu, Chenyu Du, Zijian Song, Jiayuan Song, Zhi Zhang, Yunan Huang, Weiyu Cheng, Pengyu Zhao, Yu Cheng

MaxProof is a test-time scaling framework for competition math proofs that treats one merged model as generator, verifier, refiner, and ranker, then runs an evolution-style population search (dual PATCH/REWRITE refinement, conservative min-score fitness, pairwise tournament final selection). The underlying MiniMax-M3 model is trained through three specialist stages (proof generation via CISPO RL under a four-layer generative verifier, error-finding verification, and critique-conditioned repair) before being merged into a single release model.

Zhifei Xie, Kaiyu Pang, Haobin Zhang, Deheng Ye, Xiaobin Hu, Shuicheng Yan, Chunyan Miao

Mega-ASR fine-tunes Qwen3-ASR-1.7B to handle heavily degraded real-world audio using two techniques: Acoustic-to-Semantic Progressive Supervised Fine-Tuning (a WER-graded curriculum that trains the encoder, then the LLM, then jointly) and Dual-Granularity WER-Gated Policy Optimization (a reinforcement learning reward that switches between token-level and sentence-level scoring based on WER). It is trained on Voices-in-the-Wild-2M, a 2.4M-clip synthetic dataset built by simulating 7 atomic acoustic effects and 54 compound scenarios at the spectrogram level.

Mind Lab, :, Vin Bo, Song Cao, Vic Cao, Andrew Chen, Kaijie Chen, Cleon Cheng, Steven Chiang, Kaixuan Fan, Hera Feng, Huan Feng, Arthur Fu, Jun Gao, Hongquan Gu, Aaron Guan, Nolan Ho, Mutian Hong, Hailee Hou, Peixuan Hua, Charles Huang, Miles Jiang, Nora Jiang, Yuyi Jiang, Qiuyu Jin, Fancy Kong, Andrew Lei, Kyrie Lei, Alexy Li, Lucian Li, Ray Li, Theo Li, Wenhao Li, Zhihui Li, Allen Lin, Jiayi Lin, Kairus Liu, Kieran Liu, Logan Liu, Xiang Liu, Irvine Lu, Maeve Luo, Runze Lv, Pony Ma, Verity Niu, Anson Qiu, Vincent Wang, Rio Yang, Maxwell Yao, Carrie Ye, Regis Ye, Wenlin Ye, Josh Ying, Danney Zeng, Yuhan Zhan, Anya Zhang, Di Zhang, Ruijia Zhang, Shiyang Zhang, Sueky Zhang, Ya Zhang, Wei Zhao, Ada Zhou, Adrian Zhou, Yuhua Zhou, Xinyue Zhu, Murphy Zhuang

A position paper from Mind Lab arguing that LoRA adapters should be treated as persistent per-user state on top of shared foundation models, organized around three axes: scaling up the base model (trillion-parameter MoE LoRA RL), scaling down the adapter (rank studies, a minor-subspace initialization called OLoRA-tail), and scaling out to many adapters (memory capacity laws, user simulation, majority voting). It bundles a range of experiments plus a serving/lifecycle system (MinT) that manages adapter revisions through storage, CPU cache, and GPU batch tiers.

Shuang Chen, Kaituo Feng, Hangting Chen, Wenxuan Huang, Dasen Dai, Quanxin Shou, Yunlong Lin, Xiangyu Yue, Shenghua Gao, Tianyu Pang

OpenSearch-VL is a fully open recipe (data, code, models) for training multimodal search agents that interleave reasoning with tool calls over images and web search. It combines a Wikipedia-graph-based data synthesis pipeline, a tool environment (text/image search, OCR, crop, sharpen, super-resolution, perspective correction), and a modified GRPO reinforcement learning algorithm with fatal-aware token masking and one-sided advantage clamping to salvage useful reasoning from trajectories that hit cascading tool failures.

Yuxin Zuo, Zikai Xiao, Li Sheng, Fei Huang, Jianhong Tu, Yuxuan Liu, Tianyi Tang, Xiaomeng Hu, Yang Su, Qingfeng Lan, Yantao Liu, Qin Zhu, Yinger Zhang, Bowen Yu, Haiquan Zhao, Haiyang Xu, Jianxin Yang, Jiayang Cheng, Junyang Wang, Lianghao Deng, Mingfeng Xue, Tianyi Bai, Yang Fan, Yubo Ma, Yucheng Li, Zeyu Cui, Zhihai Wang, Zhihui Xie, Zhuorui Ye, An Yang, Dayiheng Liu, Jingren Zhou, Ning Ding

Qwen-AgentWorld is a language model trained to act as a world model: given interaction history and an agent's action, it predicts the next environment observation (tool response, terminal output, UI accessibility tree, etc.) across seven domains via chain-of-thought reasoning. It is built through a three-stage pipeline (continual pre-training, supervised fine-tuning, and RL with combined rubric-and-rule rewards) and evaluated on AgentWorldBench, a benchmark of real trajectories from frontier agents.

Qiuyue Wang, Mingsheng Li, Jian Guan, Jinhui Ye, Sicheng Xie, Yitao Liu, Junhao Chen, Zhixuan Liang, Jie Zhang, Xintong Hu, Xuhong Huang, Pei Lin, Junyang Lin, Dayiheng Liu, Shuai Bai, Jingren Zhou, Jiazhao Zhang, Haoqi Yuan, Gengze Zhou, Hang Yin, Ye Wang, Yiyang Huang, Zixing Lei, Wujian Peng, Delin Chen, Yingming Zheng, Jingyang Fan, Xianwei Zhuang, Xin Zhou, Haoyang Li, Anzhe Chen, Tong Zhang, Xuejing Liu, Yuchong Sun, Ruizhe Chen, Zhaohai Li, Chenxu Lü, Zhibo Yang, Tao Yu, Xionghui Chen

Qwen-VLA is a single vision-language-action model that puts a DiT-based flow-matching action decoder on top of the Qwen3.5-4B multimodal backbone, so one model handles robot manipulation, navigation, and trajectory prediction. It uses embodiment-aware text prompts (describing the robot platform, arm config, control frequency, and prediction horizon) instead of separate output heads per robot, and is trained in four stages: text-to-action decoder pretraining without images, multimodal continued pretraining, supervised fine-tuning, and PPO reinforcement learning.

Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

SDAR trains multi-turn LLM agents by combining GRPO reinforcement learning with an auxiliary on-policy self-distillation loss, where a teacher branch is the same policy given privileged training-only context (retrieved skills). The distillation signal is filtered through a per-token sigmoid gate that strengthens supervision on tokens the teacher endorses (positive teacher-student log-probability gap) and softly attenuates tokens where the teacher disagrees.

Hongyi Liu, Haoyan Yang, Tao Jiang, Bo Tang, Feiyu Xiong, Yuyu Luo, Zhiyu Li

SkillsVote is a lifecycle framework for managing libraries of Agent Skills (structured folders of instructions, scripts, and applicability conditions used by LLM coding agents). It does three things: performs agentic search over skill folders to pick relevant skills before a task, decomposes execution traces into skill-linked subtasks after a task, and gates library updates so only successful, reusable discoveries edit or create skills.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe

This is the InstructGPT paper, which fine-tunes GPT-3 to follow instructions using reinforcement learning from human feedback (RLHF). The pipeline has three stages: supervised fine-tuning on labeler-written demonstrations, training a reward model on human rankings of model outputs, and then optimizing the policy against that reward model using PPO.