Self-Distilled Agentic Reinforcement Learning
Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
arXiv:2605.15155 · 116▲ · cs.LG, cs.AI, cs.CL
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
SDAR trains multi-turn LLM agents by combining GRPO reinforcement learning with an auxiliary on-policy self-distillation loss, where a teacher branch is the same policy given privileged training-only context (retrieved skills). The distillation signal is filtered through a per-token sigmoid gate that strengthens supervision on tokens the teacher endorses (positive teacher-student log-probability gap) and softly attenuates tokens where the teacher disagrees.
Why it matters
Naively adding distillation to agentic RL is unstable: standalone OPSD collapses and GRPO+OPSD degrades badly on smaller models because unbounded distillation gradients overwhelm the RL signal. SDAR shows a way to inject dense token-level teacher guidance without destabilizing training or biasing the RL advantage, and the resulting policy needs no retrieved skills at inference time, so it internalizes the knowledge rather than depending on skill retrieval at test time.
Practical takeaway
If you are doing RL post-training on agents and want to add distillation from a privileged-context teacher, gate the distillation loss per token by the teacher-student log-prob gap and keep the coefficient small (paper uses lambda=0.01, sigmoid sharpness beta=5) rather than distilling uniformly. Watch for the robustness finding: even random skill retrieval beat the plain GRPO baseline because the gate filters out noisy teacher signals.
Key result
On Qwen2.5-3B, SDAR improved ALFWorld success rate by +9.4% (84.4 vs 75.0), Search-QA by +7.0%, and WebShop accuracy by +4.7% over GRPO, with similar gains on 7B. Caveat: WebShop is evaluated on only 128 fixed validation tasks, training runs are 150 steps, and gains over the stronger hybrid baselines (Skill-SD, RLSD) are smaller and inconsistent on some subtasks.
Jargon
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.