DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning
Guochao Jiang, Jingyi Song, Guofeng Quan, Chuzhan Hao, Guohua Liu, Yuewei Zhang
arXiv:2605.25604 · 138▲ · cs.CL, cs.LG
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
DVAO is a method for combining multiple reward signals in GRPO-style RL training of LLMs. Instead of using fixed weights to combine rewards or advantages, it scales each objective's contribution by that objective's empirical reward standard deviation within a rollout group, up-weighting objectives that show more variation (a stronger learning signal) and down-weighting noisy or saturated ones.
Why it matters
If you train LLMs against several rewards at once (for example accuracy plus a length or output-format constraint), the usual scalarization approaches force you to hand-tune weights and often cause one easy objective to dominate or produce unstable gradients. DVAO removes the weight-tuning step and, per the paper's proofs, keeps advantage magnitudes bounded, which reduces training instability in multi-objective setups.
Practical takeaway
If you are doing multi-reward RL post-training with GRPO, you can replace fixed reward/advantage combination weights with variance-adaptive weights computed per rollout group, dropping a hyperparameter sweep. Watch the caveat: it amplifies high-variance rewards, so a noisy or poorly calibrated auxiliary reward can get wrongly up-weighted.
Key result
On tool-use (BFCL-v4, Qwen2.5-7B-Instruct) DVAO reached 63.00% average accuracy and 79.21% format compliance, versus 60.13%/68.12% for GDPO and 44.25%/68.04% for advantage combination. Evaluations are limited to dual-objective settings (accuracy plus length or format) with equal-weight initialization across all baselines; multi-objective (3+ reward) behavior is untested.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.