CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence
Dongsheng Ma, Jiayu Li, Zhengren Wang, Yijie Wang, Jiahao Kong, Weijun Zeng, Jutao Xiao, Jie Yang, Wentao Zhang, Bin Wang, Conghui He
arXiv:2605.12882 · 274▲ · cs.CL, cs.CV
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
CiteVQA is a document VQA benchmark that requires models to return element-level bounding-box citations alongside each answer, then scores both jointly with a metric called Strict Attributed Accuracy (SAA), which only credits a sample when the answer is correct AND the cited region matches ground truth. Ground-truth citations are generated by an automated pipeline (document parsing plus MLLM agents plus masking-based ablation to find crucial evidence) and validated by expert review across 1,897 questions from 711 multi-page PDFs.
Why it matters
If you are building document-processing systems for law, finance, or medicine, answer-only accuracy hides a failure mode the authors call Attribution Hallucination: the model gives the right answer while pointing to the wrong source region, which breaks any workflow that needs traceable, auditable evidence. This benchmark gives you a way to measure that gap rather than trusting a correct-looking answer.
Practical takeaway
When evaluating an MLLM for document retrieval or RAG-style grounding, test whether it can localize the exact evidence box, not just answer correctly, because even strong models fail at page-level navigation. Expect open-source models to be unreliable here for now, and note that narrowing the search space (feeding only gold pages or documents) measurably improves both grounding and answers.
Key result
The best system, Gemini-3.1-Pro-Preview, reached only 76.0 SAA overall while scoring 86.1 on answer correctness, and the best open-source model (Qwen3-VL-235B) hit just 22.5 SAA. Note answer and relevance scores come from an LLM judge (Qwen3-VL-235B), and recall uses a lenient IoU 0.5 threshold, so the absolute numbers depend on those judge and threshold choices.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.