EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions
Jincheng Zhong, Weizhi Wang, Che Jiang, Kai Tian, Zhenzhao Yuan, Junlin Yang, Dianqiao Lei, Kaiyan Zhang
arXiv:2606.23654 · 79▲ · cs.CL, cs.SE
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
EnterpriseClawBench is a benchmark built by an automated pipeline that converts real internal workplace agent sessions into 852 reproducible tasks, each with recovered input fixtures, rewritten single-turn prompts, role/skill taxonomy labels, hard delivery rules, and semantic rubrics. It evaluates harness-model combinations (Claude Code, Codex, DeepAgents, Hermes, OpenClaw paired with various models) on producing usable business artifacts, scoring both objective delivery rules and LLM-judged quality across five dimensions.
Why it matters
It shows that agent performance in real enterprise work depends heavily on the harness wrapping the model, not just the model itself: the same Claude model can succeed under one harness and fail to write a stable output file under another. For anyone deploying agents, this means you must benchmark specific harness-model pairs plus cost, runtime, and artifact delivery rather than trusting a single model leaderboard score.
Practical takeaway
When picking an agent stack, test the exact harness-model combination on your own tasks and watch for harness-level behaviors (approval gates, subtask delegation, truncated traces) that silently prevent artifacts from landing in the output directory. Also treat multimodal LLM judges with suspicion: they inflate scores on spreadsheets and slides and disagree with humans on visual output.
Key result
The best configuration (Codex with GPT-5.5) scored only 0.663 on the manually audited 120-task Lite subset, showing the tasks are far from saturated. A key caveat on the evaluation itself: the visual judge (Sonnet 4.6) had MAE 0.303 and negative Spearman correlation (-0.259) against human raters on 24 visual packets, so visual scores are unreliable, while the text route agreed well (MAE 0.134, Spearman 0.790). The benchmark data is not released because it contains proprietary sessions.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.