DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards
Kaiyi Zhang, Wei Wu, Yankai Lin
arXiv:2605.21467 · 207▲ · cs.LG, cs.CL
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
The paper reframes RLVR (reinforcement learning from verifiable rewards) policy updates as an implicit linear discriminator over token-gradient vectors, arguing that standard sequence-level methods like DAPO let shared high-frequency tokens (formatting, entities) dominate the update direction. It proposes DelTA, which computes per-token coefficients based on how well each token's gradient distinguishes positive-advantage from negative-advantage responses, then uses those coefficients to reweight the DAPO surrogate loss.
Why it matters
For teams running RLVR to improve model reasoning, this is a drop-in modification to the training objective that reweights token contributions without needing a critic, process reward model, or external dense reward signal. The coefficients are computed once per rollout batch using a cheap LM-head gradient proxy, adding roughly 10% to first-step time, so the overhead is modest relative to rollout generation.
Practical takeaway
If you already train with DAPO or a similar group-relative RLVR method, you can try reweighting the per-token loss by a discriminative coefficient (bounded to roughly [0.8, 1.2]) rather than treating all tokens in a response equally. Watch for the finding that training on only the top 50% of tokens by this coefficient beats full-token training, while the bottom 50% causes collapse, suggesting some token gradients actively hurt learning.
Key result
On seven math benchmarks with Qwen3-8B-Base, DelTA raised the average score from 25.14 (strongest same-scale baseline, SAPO) to 28.40, a 3.26-point gain; on Qwen3-14B-Base it improved from 37.29 (FIPO) to 39.91. Gains were consistent but modest across all benchmarks, and were reproduced on Olmo3-7B-Base (+3.79 avg) and code generation (47.7 to 49.5 weighted avg).
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.