NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
Yuru Wang, Lejun Cheng, Yuxin Zuo, Sihang Zeng, Bingxiang He, Che Jiang, Junlin Yang, Yuchong Wang, Kaikai Zhao, Weifeng Huang, Kai Tian, Zhenzhao Yuan, Jincheng Zhong, Weizhi Wang, Ning Ding, Bowen Zhou, Kaiyan Zhang
arXiv:2606.24530 · 62▲ · cs.CL
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
NatureBench is a benchmark of 90 tasks distilled from Nature-family science papers (2022-2025), built by an automated pipeline (NatureGym) that turns each paper into a containerized package with a task brief, data, a hidden test set, and an automated evaluator, while removing the original method so agents must find their own solution. It scores coding agents against each paper's published state of the art using a SOTA-normalized relative gap metric, with a post-hoc judge that flags shortcuts like output fabrication and feedback gaming.
Why it matters
For anyone tracking whether AI coding agents can do real scientific research rather than reproduce known code, this gives a concrete, reproducible measurement: current frontier agents rarely beat published results and mostly succeed by recasting science problems as standard supervised ML pipelines. This is mainly a signal for practitioners evaluating agent hype, since it shows the bottleneck is method selection and compute depth, not code generation.
Practical takeaway
Watch for agent claims of scientific discovery with skepticism: this benchmark shows agents win by turning tasks into generic prediction problems, not by inventing methods. If you deploy coding agents on hard technical tasks, expect wrong method choice and insufficient compute budget to be the dominant failure modes, not misunderstanding the task or broken code.
Key result
The strongest agent, Claude Opus 4.7, surpassed the published SOTA (relative gap g>0.1) on only 17.8% of the 90 tasks and matched it (g>=0) on 47.8%, under a 4-hour, web-search-disabled, single-GPU protocol; across all 900 runs, 45.5% of successes came from supervised proxy prediction rather than domain-specific methods.
Jargon
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.