Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent
Lei Bai, Zongsheng Cao, Yang Chen, Zhiyao Cui, Shangheng Du, Yue Fan, Shiyang Feng, Zijie Guo, Haonan He, Liang He, Xiaohan He, Shuyue Hu, Yusong Hu, Songtao Huang, Yichen Jiang, Hao Li, Xin Li, Dahua Lin, Weihao Lin, Fenghua Ling, Dongrui Liu, Zhuo Liu, Runmin Ma, Chunjiang Mu, Haoyang Peng, Tianshuo Peng, Jinxin Shi, Luohe Shi, Boyuan Sun, Zelin Tan, Shengji Tang, Qianyi Wang, Yiming Wu, Yi Xie, Xiangchao Yan, Jingqi Ye, Peng Ye, Fangchen Yu, Jiakang Yuan, Bihao Zhan, Bo Zhang, Chen Zhang, Shufei Zhang, Shuaiyu Zhang, Wenlong Zhang, Yiqun Zhang, Junpeng Zhao, Zhijie Zhong, Bowen Zhou, Yuhao Zhou
arXiv:2606.30616 · 90▲ · cs.CL
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
A recipe for training a 35B mixture-of-experts agent (Agents-A1) to handle long-horizon agentic tasks by scaling trajectory length rather than parameters. The method has three stages: full-domain supervised fine-tuning on ~100K long trajectories (average 45K tokens), training domain-specific teacher models, then consolidating them into one student via domain-routed on-policy distillation with salient vocabulary alignment (SVA), which aligns student and teacher on the teacher's top-k token support instead of just the sampled token.
Why it matters
It argues you can reach frontier-level agentic performance in narrow domains (search, science, coding, tool use) without a trillion-parameter model, which lowers the compute and serving cost of deploying capable agents. The training infrastructure (a knowledge-action graph that records evidence, actions, observations, and verifier outcomes) is the reusable idea for anyone building verifiable multi-step agent training data.
Practical takeaway
Watch for smaller MoE models tuned specifically for long-horizon tool use as an alternative to calling giant frontier APIs. The distillation trick (aligning on the teacher's top-k salient vocabulary and hard-routing each sample to its domain teacher) is a concrete fix for the instability of naive on-policy distillation across mixed domains.
Key result
The paper claims Agents-A1 (35B) matches or beats 1T-parameter models (Kimi-K2.6, DeepSeek-V4) on SEAL-0, IFBench, HiPhO, FrontierScience-Olympiad, and MolBench-Bind, with strong results on SciCode, HLE, and BrowseComp. Caveat: the summary figure and text report relative comparisons on selected science and search benchmarks rather than head-to-head numbers across a broad general-purpose suite, so the parity claim is scoped to these curated domains.
Jargon
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.