OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents
Shuang Chen, Kaituo Feng, Hangting Chen, Wenxuan Huang, Dasen Dai, Quanxin Shou, Yunlong Lin, Xiangyu Yue, Shenghua Gao, Tianyu Pang
arXiv:2605.05185 · 106▲ · cs.CV
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
OpenSearch-VL is a fully open recipe (data, code, models) for training multimodal search agents that interleave reasoning with tool calls over images and web search. It combines a Wikipedia-graph-based data synthesis pipeline, a tool environment (text/image search, OCR, crop, sharpen, super-resolution, perspective correction), and a modified GRPO reinforcement learning algorithm with fatal-aware token masking and one-sided advantage clamping to salvage useful reasoning from trajectories that hit cascading tool failures.
Why it matters
Strong multimodal deep-search agents have mostly been proprietary with undisclosed training data and recipes. This release gives practitioners a reproducible foundation to build agents that identify visual entities, repair low-quality images, and chain multi-hop web retrieval, rather than relying on closed commercial APIs.
Practical takeaway
If you are building an agent that answers knowledge-intensive questions about images, you can now start from released Qwen3-VL-based checkpoints and the SearchVL datasets instead of synthesizing your own tool-use trajectories. Watch for the data curation tricks (fuzzy entity rewriting, anchoring the visual entry point away from the answer) as a way to stop agents from short-circuiting multi-hop questions with a single lookup.
Key result
OpenSearch-VL-30B-A3B raises average Pass@1 from 47.8 to 61.6 across seven benchmarks versus its Qwen3-VL-30B-A3B agentic baseline, with the largest single-benchmark gain of +24.5 on MMSearch. Correctness is judged by GPT-4o rather than exact match, and the strongest proprietary comparisons are mostly direct-reasoning models rather than equivalently tooled agents.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.