metadata

base_model:
  - Qwen/Qwen2-VL-7B-Instruct
language:
  - en
license: apache-2.0
metrics:
  - accuracy
pipeline_tag: image-text-to-text
library_name: transformers

DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs for Knowledge-Intensive Visual Grounding

This is the official repository of DeepPerception, an MLLM enhanced with cognitive visual perception capabilities.

Project Page

Paper

Overview

Figure 1: (a) DeepPerception employs knowledge-driven reasoning to derive answers, while the baseline model directly outputs predictions without cognitive processing. (b) DeepPerception demonstrates superior cognitive visual perception capabilities that cannot be elicited in the foundation model through simplistic zero-shot CoT prompting.

Abstract

Human experts excel at fine-grained visual discrimination by leveraging domain knowledge to refine perceptual features, a capability that remains underdeveloped in current Multimodal Large Language Models (MLLMs). Despite possessing vast expert-level knowledge, MLLMs struggle to integrate reasoning into visual perception, often generating direct responses without deeper analysis.

To bridge this gap, we introduce knowledge-intensive visual grounding (KVG), a novel visual grounding task that requires both finegrained perception and domain-specific knowledge integration. To address the challenges of KVG, we propose DeepPerception, an MLLM enhanced with cognitive visual perception capabilities. Our approach consists of (1) an automated data synthesis pipeline that generates high-quality, knowledge-aligned training samples, and (2) a two-stage training framework combining supervised fine-tuning for cognitive reasoning scaffolding and reinforcement learning to optimize perceptioncognition synergy. To benchmark performance, we introduce KVG-Bench, a comprehensive dataset spanning 10 domains with 1.3K manually curated test cases.

Experimental results demonstrate that DeepPerception significantly outperforms direct fine-tuning, achieving +8.08% accuracy improvements on KVG-Bench and exhibiting +4.60% superior cross-domain generalization over baseline approaches. Our findings highlight the importance of integrating cognitive processes into MLLMs for human-like visual perception and open new directions for multimodal reasoning research.

Key Contributions

We introduce the task of Knowledge-intensive Visual Grounding (KVG) to explore the concept of cognitive visual perception for MLLMs, aiming to integrate their inherent knowledge and reasoning capabilities into visual perception.
We propose DeepPerception, an MLLM with enhanced cognitive visual perception capabilities. To achieve this, we develop an automated dataset creation pipeline and a two-stage framework integrating supervised cognitive capability enhancement with perception-oriented reinforcement learning.
We introduce KVG-Bench, a manually curated benchmark for the KVG task involving diverse knowledge domains and entities. Experiments on KVG-Bench and other fine-grained visual recognition tasks demonstrate DeepPerception's exceptional cognitive visual perception capabilities and superior cross-domain generalization performance.

Get Started

Environment

Clone this repository and navigate to DeepPerception folder

git clone https://github.com/MaxyLee/DeepPerception.git
cd DeepPerception

Install Packages For evaluation:

conda env create -n deepperception python=3.9
conda activate deepperception

pip install -r requirements.txt

Data Preparation

Dataset	Links
KVG-Bench	`🤗HuggingFace`
KVG Training	`🤗HuggingFace`

Checkpoints

Model	Links
DeepPerception	`🤗HuggingFace`
DeepPerception-FGVR	`🤗HuggingFace`

Evaluation

# Evaluate on KVG-Bench
bash eval.sh [CUDA_IDS] [KVG_BENCH_PATH] [CKPT_PATH]

Notice: Please modify the script if you want to evaluate on Qwen2-VL.

Training

TODO

Citation

If you find DeepPerception useful for your research or applications, please cite using this BibTeX:

@misc{ma2025deepperception,
      title={DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs for Knowledge-Intensive Visual Grounding}, 
      author={Xinyu Ma and Ziyang Ding and Zhicong Luo and Chi Chen and Zonghao Guo and Derek F. Wong and Xiaoyi Feng and Maosong Sun},
      year={2025},
      url={https://arxiv.org/abs/2503.12797}, 
}

MaxyLee
/

DeepPerception-FGVR