arxiv:2508.13186

MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents

Published on Aug 14

· Submitted by

sefira32 on Aug 20

Upvote

Authors:

Xingyuan Bu ,

Abstract

A new benchmark, MM-BrowseComp, evaluates AI agents' multimodal retrieval and reasoning capabilities, revealing limitations in current models' handling of images and videos in web browsing tasks.

AI-generated summary

AI agents with advanced reasoning and tool use capabilities have demonstrated impressive performance in web browsing for deep search. While existing benchmarks such as BrowseComp evaluate these browsing abilities, they primarily focus on textual information, overlooking the prevalence of multimodal content. To bridge this gap, we introduce MM-BrowseComp, a novel benchmark comprising 224 challenging, hand-crafted questions specifically designed to assess agents' multimodal retrieval and reasoning capabilities. These questions often incorporate images in prompts, and crucial information encountered during the search and reasoning process may also be embedded within images or videos on webpages. Consequently, methods relying solely on text prove insufficient for our benchmark. Additionally, we provide a verified checklist for each question, enabling fine-grained analysis of multimodal dependencies and reasoning paths. Our comprehensive evaluation of state-of-the-art models on MM-BrowseComp reveals that even top models like OpenAI o3 with tools achieve only 29.02\% accuracy, highlighting the suboptimal multimodal capabilities and lack of native multimodal reasoning in current models.

View arXiv page View PDF Project page GitHub 14 Add to collection

Community

sefira32

Paper author Paper submitter 2 days ago

•

edited 2 days ago

MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents, repo at https://github.com/MMBrowseComp/MM-BrowseComp

sefira32

Paper author Paper submitter 2 days ago

•

edited 1 day ago

Is text-only information enough for LLM/VLM Web Agents? 🤔 Clearly not. 🙅‍♂️ The modern web is a rich tapestry of text, images 🖼️, and videos 🎥. To truly assist us, agents need to understand it all. That's why we built MM-BrowseComp. 🌐

We're introducing MM-BrowseComp 🚀, a new benchmark designed to push web agents beyond text. It features 224 handcrafted tasks ✍️ that require agents to not just read, but also see 👀 and comprehend multimodal content to find answers.

Dataset & Code @GitHub: https://github.com/MMBrowseComp/MM-BrowseComp
Huggingface: https://huggingface.co/datasets/mmbrowsecomp/MMBrowseComp
arXiv Page: https://www.arxiv.org/abs/2508.13186
Daily Paper: https://huggingface.co/papers/2508.13186

librarian-bot

1 day ago

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

sefira32

Paper author Paper submitter 1 day ago

[1/6]
Our headline finding? 🤯 Even the most powerful models aren't ready for the multimodal web. Top-performing models like OpenAI's o3 with tools only reached an accuracy of 29.02% 📉, showing a significant gap in their ability to handle the visual web. 🤖🕸️

[2/6]
Current models struggle specifically with multimodal content. Our fine-grained analysis shows that agents perform significantly worse when the information is in images 🖼️➡️😩 or videos 🎥➡️😵. Their multimodal capabilities are simply not up to par.

[3/6]
Today's agents are not true multimodal reasoners. 🧠 They often rely on separate captioning tools to understand images 🖼️➡️💬, which leads to information loss and sometimes even made-up details. This highlights a need for models with integrated, native multimodal reasoning. 💡

[4/6]
Agents that can "reflect" on their actions perform better. 🧠✨ We found that architectures using reflection and ReAct-style mechanisms are more robust. 💪 They can recover from errors and don't blindly trust their tools—a crucial skill for complex web tasks. ✅

[5/6]
Success requires a double threat: strong reasoning and a complete toolset. 🏆 Our evaluation shows that models excelling in just one area fall short. The best performance comes from the synergy of a powerful reasoning engine 🧠 and a comprehensive set of tools 🛠️. 🤝

[6/6]
Simply giving agents more tries doesn't fix the core problem. 🔁 Our analysis shows that increasing test-time attempts provides only marginal gains. This suggests the primary bottleneck is a fundamental lack of reasoning ability, not just bad luck on the first try. 🎲❌