Add comprehensive model card for MLLMSeg_InternVL2_5_4B_RES (#1)

- Add comprehensive model card for MLLMSeg_InternVL2_5_4B_RES (30eed674e10caeef4200bcfa974f2f78e7459578)


Co-authored-by: Niels Rogge <[email protected]>

README.md

@@ -1,3 +1,176 @@
----
-license: mit
----
+---
+license: mit
+pipeline_tag: image-segmentation
+library_name: transformers
+---
+
+# MLLMSeg: Unlocking the Potential of MLLMs in Referring Expression Segmentation via a Light-weight Mask Decoder
+
+This repository contains the `MLLMSeg_InternVL2_5_4B_RES` model, which was presented in the paper [Unlocking the Potential of MLLMs in Referring Expression Segmentation via a Light-weight Mask Decoder](https://huggingface.co/papers/2508.04107).
+
+**MLLMSeg** is a novel framework designed for Referring Expression Segmentation (RES). It fully exploits the inherent visual detail features encoded in Multimodal Large Language Model (MLLM) vision encoders without introducing an extra visual encoder. The model employs a detail-enhanced and semantic-consistent feature fusion module (DSFF) and a light-weight mask decoder (with only 34M network parameters) to achieve precise mask prediction. This approach strikes a better balance between performance and cost compared to existing methods.
+
+Code: https://github.com/jcwang0602/MLLMSeg
+
+## Quick Start (Inference)
+
+### Installation
+
+First, install the `transformers` library and other dependencies. For a complete installation guide, please refer to the [official GitHub repository](https://github.com/jcwang0602/MLLMSeg) and the [InternVL2 documentation](https://internvl.readthedocs.io/en/latest/get_started/installation.html).
+
+```bash
+conda create -n mllmseg python==3.10.18 -y
+conda activate mllmseg
+pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu118
+pip install -r requirements.txt
+pip install flash-attn==2.3.6 --no-build-isolation # Note: need gpu to install
+```
+
+### Inference Example
+
+Here's an example to perform inference with the model:
+
+```python
+import numpy as np
+import torch
+import torchvision.transforms as T
+from PIL import Image
+from torchvision.transforms.functional import InterpolationMode
+from transformers import AutoModel, AutoTokenizer
+
+IMAGENET_MEAN = (0.485, 0.456, 0.406)
+IMAGENET_STD = (0.229, 0.224, 0.225)
+
+def build_transform(input_size):
+    MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
+    transform = T.Compose([
+        T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
+        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
+        T.ToTensor(),
+        T.Normalize(mean=MEAN, std=STD)
+    ])
+    return transform
+
+def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
+    best_ratio_diff = float('inf')
+    best_ratio = (1, 1)
+    area = width * height
+    for ratio in target_ratios:
+        target_aspect_ratio = ratio[0] / ratio[1]
+        ratio_diff = abs(aspect_ratio - target_aspect_ratio)
+        if ratio_diff < best_ratio_diff:
+            best_ratio_diff = ratio_diff
+            best_ratio = ratio
+        elif ratio_diff == best_ratio_diff:
+            if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
+                best_ratio = ratio
+    return best_ratio
+
+def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):
+    orig_width, orig_height = image.size
+    aspect_ratio = orig_width / orig_height
+
+    # calculate the existing image aspect ratio
+    target_ratios = set(
+        (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
+        i * j <= max_num and i * j >= min_num)
+    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])
+
+    # find the closest aspect ratio to the target
+    target_aspect_ratio = find_closest_aspect_ratio(
+        aspect_ratio, target_ratios, orig_width, orig_height, image_size)
+
+    # calculate the target width and height
+    target_width = image_size * target_aspect_ratio[0]
+    target_height = image_size * target_aspect_ratio[1]
+    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]
+
+    # resize the image
+    resized_img = image.resize((target_width, target_height))
+    processed_images = []
+    for i in range(blocks):
+        box = (
+            (i % (target_width // image_size)) * image_size,
+            (i // (target_width // image_size)) * image_size,
+            ((i % (target_width // image_size)) + 1) * image_size,
+            ((i // (target_width // image_size)) + 1) * image_size
+        )
+        # split the image
+        split_img = resized_img.crop(box)
+        processed_images.append(split_img)
+    assert len(processed_images) == blocks
+    if use_thumbnail and len(processed_images) != 1:
+        thumbnail_img = image.resize((image_size, image_size))
+        processed_images.append(thumbnail_img)
+    return processed_images
+
+def load_image(image_file, input_size=448, max_num=12):
+    image = Image.open(image_file).convert('RGB')
+    transform = build_transform(input_size=input_size)
+    images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
+    pixel_values = [transform(image) for image in images]
+    pixel_values = torch.stack(pixel_values)
+    return pixel_values
+
+# Load the model and tokenizer
+path = 'jcwang0602/MLLMSeg_InternVL2_5_4B_RES'
+model = AutoModel.from_pretrained(
+    path,
+    torch_dtype=torch.bfloat16,
+    low_cpu_mem_usage=True,
+    trust_remote_code=True).eval().cuda()
+tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)
+
+# Prepare image and question (replace './examples/images/web_dfacd48d-d2c2-492f-b94c-41e6a34ea99f.png' with your image path)
+pixel_values = load_image('./examples/images/web_dfacd48d-d2c2-492f-b94c-41e6a34ea99f.png', max_num=6).to(torch.bfloat16).cuda()
+generation_config = dict(max_new_tokens=1024, do_sample=True)
+
+question = "In the screenshot of this web page, please give me the coordinates of the element I want to click on according to my instructions(with point).\
+\\\"'Champions League' link\\\""
+
+# Chat with the model
+response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)
+print(f'User: {question}\
+Assistant: {response}')
+```
+
+## Performance Metrics
+
+The following tables showcase the performance of MLLMSeg on various benchmarks, as presented in the original repository:
+
+### Referring Expression Segmentation
+<img src="https://github.com/jcwang0602/MLLMSeg/raw/main/assets/tab_res.png" width="800">
+
+### Referring Expression Comprehension
+<img src="https://jcwang0602/MLLMSeg/raw/main/assets/tab_rec.png" width="800">
+
+### Generalized Referring Expression Segmentation
+<img src="https://github.com/jcwang0602/MLLMSeg/raw/main/assets/tab_gres.png" width="800">
+
+## Visualization
+
+Visual examples of MLLMSeg's performance:
+
+### Referring Expression Segmentation
+<img src="https://github.com/jcwang0602/MLLMSeg/raw/main/assets/res.png" width="800">
+
+### Referring Expression Comprehension
+<img src="https://github.com/jcwang0602/MLLMSeg/raw/main/assets/rec.png" width="800">
+
+### Generalized Referring Expression Segmentation
+<img src="https://github.com/jcwang0602/MLLMSeg/raw/main/assets/gres.png" width="800">
+
+## Citation
+
+If our work is useful for your research, please consider citing:
+```bibtex
+@misc{wang2025unlockingpotentialmllmsreferring,
+      title={Unlocking the Potential of MLLMs in Referring Expression Segmentation via a Light-weight Mask Decoder}, 
+      author={Jingchao Wang and Zhijian Wu and Dingjiang Huang and Yefeng Zheng and Hong Wang},
+      year={2025},
+      eprint={2508.04107},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV},
+      url={https://arxiv.org/abs/2508.04107}, 
+}
+```