K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts
Nahyun Lee, Dongkeun Yoon, Guijin Son, Geewook Kim, Dayoon Ko, Jeonghun Park, Haneul Yoo, Jaewon Cho, Junghun Park, Changyoon Lee, Kyochul Jang, Jaeyeon Kim, Eunsu Kim, Woojin Cho, Seungone Kim
arXiv:2606.02404 · 59▲ · cs.CL
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
K-BrowseComp is a web-browsing agent benchmark of 400 problems grounded in Korean web contexts: a 300-question human-verified subset built by native Korean annotators, plus a 100-question synthetic split generated by a browsing agent (Claude Code) that constructs questions backwards from seed pages and filters them against a taxonomy of failure modes. Each question has a single stable answer requiring multi-hop or parallel-constraint retrieval across multiple Korean websites, and models are evaluated pass@1 under a fixed 10-call search budget.
Why it matters
The benchmark shows that browsing-agent performance does not transfer across language and cultural contexts: frontier models that score in the 80s on the English BrowseComp collapse to 30 to 46 percent here, and Korean government-funded models score near zero. For anyone building agents for non-English markets, this quantifies that broad language coverage and model scale do not fix the real bottleneck, which is maintaining candidate sets, constraints, and entity roles across a multi-step trajectory after evidence is already retrieved.
Practical takeaway
If you deploy browsing agents in a specific locale, test them on locale-specific retrieval rather than trusting English benchmark scores, and watch trajectory-level state handling (candidate tracking, constraint merging, answer finalization) rather than just adding more search calls. The paper also demonstrates you can use an LLM agent to auto-generate hard evaluation items by targeting known failure modes, so expect more failure-mode-driven synthetic benchmark construction in tooling.
Key result
On the verified subset the best model, GPT-5.5, reached 45.67 percent versus its reported 84.4 percent on English BrowseComp, while Korean open-weight models scored 0.00 to 10.33 percent; on the adversarially filtered synthetic split the top score was 26.00 percent. Caveat: results come from a single browsing harness, one search backend, and single-run pass@1, and the synthetic split is separately reported because failure-based filtering (against gpt-5.4-mini and gemini-3-flash-preview) is baked into its construction.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.