|
|
--- |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- antgroup/HumanSense_Benchmark |
|
|
language: |
|
|
- en |
|
|
metrics: |
|
|
- accuracy |
|
|
library_name: transformers |
|
|
base_model: |
|
|
- Qwen/Qwen2.5-Omni-7B |
|
|
pipeline_tag: video-text-to-text |
|
|
--- |
|
|
|
|
|
|
|
|
<div align="center" style="font-family: charter;"> |
|
|
|
|
|
|
|
|
<p align="center"> |
|
|
<img src="pic.png" width="400"/> |
|
|
<p> |
|
|
|
|
|
<!-- <h1></br>From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMs</h1> --> |
|
|
|
|
|
<div> |
|
|
<a href="https://scholar.google.com/citations?user=sPQqpXsAAAAJ&hl=en&oi=sra">Zheng Qin<sup>1</sup></a>, |
|
|
<a href="https://scholar.google.com/citations?user=S8FmqTUAAAAJ&hl=en">Ruobing Zheng<sup>*</sup><sup>2</sup></a>, |
|
|
<a href="https://scholar.google.com/citations?user=3WVFdMUAAAAJ&hl=en">Yabing Wang<sup>1</sup></a>, |
|
|
<a href="https://scholar.google.com/citations?user=yOtsVWQAAAAJ&hl=en&oi=sra">Tianqi Li<sup>2</sup></a>, |
|
|
<a href="https://yuanyi.pub/">Yi Yuan<sup>2</sup></a>, |
|
|
<a href="https://scholar.google.com/citations?hl=en&user=8SCEv-YAAAAJ&view_op=list_works&sortby=pubdate">Jingdong Chen<sup>2</sup></a>, |
|
|
<a href="https://scholar.google.com/citations?user=RypRCUQAAAAJ&hl=en">Le Wang<sup>†<dag><sup>1</sup></a> <br> |
|
|
<span style="font-size: 13px; margin-top: 0.8em"> |
|
|
<br> |
|
|
<sup>*</sup>Co-first authors. Project Lead. |
|
|
<sup>†</sup>Corresponding Author. |
|
|
<br> |
|
|
<sup>1</sup>Xi’an Jiaotong University. <sup>2</sup>Ant Group. |
|
|
<br> |
|
|
</span> |
|
|
</div> |
|
|
|
|
|
|
|
|
<a target="_blank" href="https://arxiv.org/abs/2508.10576" ><button><i class="ai ai-arxiv"></i> arXiv:2508.10576</button></a> |
|
|
|
|
|
<a target="_blank" href="https://digital-avatar.github.io/ai/HumanSense/" ><button><i class="ai ai-arxiv"></i> Homepage</button></a> |
|
|
|
|
|
<a target="_blank" href="https://github.com/antgroup/HumanSense" ><button><i class="ai ai-arxiv"></i> GitHub</button></a> |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<img src="figure1.png" width="100%"/> |
|
|
<p align="justify"><i>While Multimodal Large Language Models (MLLMs) show immense promise for achieving truly human-like interactions, progress is hindered by the lack of fine-grained evaluation frameworks for human-centered scenarios, encompassing both the understanding of complex human intentions and the provision of empathetic, context-aware responses. Here we introduce <strong>HumanSense</strong>, a comprehensive benchmark designed to evaluate the human-centered perception and interaction capabilities of MLLMs, with a particular focus on deep understanding of extended multimodal contexts and the formulation of rational feedback. Our evaluation reveals that leading MLLMs still have considerable room for improvement, particularly for advanced interaction-oriented tasks. Supplementing visual input with audio and text information yields substantial improvements, and Omni-modal models show advantages on these tasks. Furthermore, we argue that appropriate feedback stems from a contextual analysis of the interlocutor's needs and emotions, with reasoning ability serving as the key to unlocking it. Accordingly, we devise a multi-stage, modality-progressive reinforcement learning approach, resulting in <strong>HumanSense-Omni-Reasoning</strong>, which substantially enhances performance on higher-level understanding and interactive tasks. Additionally, we observe that successful reasoning processes exhibit highly consistent thought patterns. By designing corresponding prompts, we also enhance the performance of non-reasoning models in a training-free manner. |
|
|
</i></p> |
|
|
|
|
|
</div> |
|
|
|
|
|
## Release |
|
|
- `2025-08-27` :hearts: We release both the training code and dataset! |
|
|
- `2025-08-27` :hearts: We released Benchmark and code! |
|
|
- `2025-08-15` :rocket: We released our paper! |
|
|
|
|
|
|
|
|
|
|
|
## Quickstart |
|
|
|
|
|
Below, we provide simple examples to show how to use HumanSense_Omni_Reasoning with 🤗 Transformers. |
|
|
``` |
|
|
pip uninstall transformers |
|
|
pip install transformers==4.52.0 |
|
|
pip install accelerate |
|
|
pip install qwen-omni-utils |
|
|
pip install qwen-omni-utils[decord] -U |
|
|
``` |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor |
|
|
from qwen_omni_utils import process_mm_info |
|
|
|
|
|
|
|
|
model_path = "antgroup/HumanSense_Omni_Reasoning" |
|
|
|
|
|
|
|
|
model = Qwen2_5OmniForConditionalGeneration.from_pretrained( |
|
|
model_path, |
|
|
torch_dtype=torch.bfloat16, |
|
|
device_map="auto", |
|
|
attn_implementation="flash_attention_2", |
|
|
) |
|
|
model.disable_talker() |
|
|
processor = Qwen2_5OmniProcessor.from_pretrained(model_path) |
|
|
|
|
|
conversation = [ |
|
|
{ |
|
|
"role": "user", |
|
|
"content": [ |
|
|
{ |
|
|
"type": "video", |
|
|
"video": "file:///path/to/xxx.mp4", |
|
|
"max_pixels": 151200 |
|
|
}, |
|
|
{ |
|
|
"type": "text", |
|
|
"text": "xxxxxxxxxxxxxxxxxx\n" |
|
|
} |
|
|
], |
|
|
} |
|
|
] |
|
|
|
|
|
USE_AUDIO_IN_VIDEO=True |
|
|
|
|
|
text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False) |
|
|
audios, images, videos = process_mm_info(conversation, use_audio_in_video=USE_AUDIO_IN_VIDEO) |
|
|
inputs = processor(text=text, audio=audios, images=images, videos=videos, return_tensors="pt", padding=True,padding_side="left",add_special_tokens=False, use_audio_in_video=USE_AUDIO_IN_VIDEO) |
|
|
inputs = inputs.to(model.device).to(model.dtype) |
|
|
|
|
|
# Inference: Generation of the output text and audio |
|
|
text_ids = model.generate(**inputs,return_audio=False, use_audio_in_video=USE_AUDIO_IN_VIDEO) |
|
|
generated_ids_trimmed = [ |
|
|
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, text_ids) |
|
|
] |
|
|
text = processor.batch_decode(generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False) |
|
|
|
|
|
|
|
|
|
|
|
response = text[0] |
|
|
print('*'*30) |
|
|
print(response) |
|
|
|
|
|
``` |
|
|
|
|
|
<p align="justify"><i>Examples of Reasoning: </i></p> |
|
|
<img src="figure5.png" width="100%"/> |
|
|
<p align="justify"><i>These cases cover four high-level perception and interaction tasks, including both video-based and audio-based questions. The reasoning processes all demonstrate thinking that integrates characteristics, emotions, and context, and then provides appropriate feedback. |
|
|
</i></p> |
|
|
|
|
|
</div> |
|
|
|
|
|
|
|
|
|
|
|
**BibTeX:** |
|
|
|
|
|
``` |
|
|
@article{qin2025humansense, |
|
|
title={HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMs}, |
|
|
author={Qin, Zheng and Zheng, Ruobing and Wang, Yabing and Li, Tianqi and Yuan, Yi and Chen, Jingdong and Wang, Le}, |
|
|
journal={arXiv preprint arXiv:2508.10576}, |
|
|
year={2025} |
|
|
} |
|
|
``` |
