docs/LLaVA-NeXT-Interleave.md

# LLaVA-NeXT: Tackling Multi-image, Video, and 3D in Large Multimodal Models

## Contents
- [Demo](#demo)
- [Evaluation](#evaluation)

## Demo

> make sure you installed the LLaVA-NeXT model files via outside REAME.md

1. **Example model:** `lmms-lab/llava-next-interleave-7b`


To run a demo, execute:
```bash
# If you find any bug when running the demo, please make sure checkpoint path contains 'qwen'.
# You can try command like 'mv llava-next-interleave-7b llava-next-interleave-qwen-7b'
python playground/demo/interleave_demo.py --model_path path/to/ckpt
```

## Evaluation

### Preparation

Please download the evaluation data and its metadata from the following links:

1. **llava-interleave-bench:** [here](https://huggingface.co/datasets/lmms-lab/llava-interleave-bench).

Unzip eval_images.zip and there are Split1 and Split2 in it.
Organize the downloaded data into the following structure:
```

interleave_data
├── Split1
│   ├── ...
│   └── ...
|
├── Split2
|   ├── ...
│   └── ...
├── multi_image_in_domain.json
├── multi_image_out_domain.json
└── multi_view_in_domain.json
```

### Inference and Evaluation
Example:
Please first edit /path/to/ckpt to the path of checkpoint, /path/to/images to the path of "interleave_data" in scripts/interleave/eval_all.sh and then run
```bash
bash scripts/interleave/eval_all.sh
```