initial commit

README.md

@@ -1,3 +1,94 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+datasets:
+- AIDC-AI/Ovis-dataset
+library_name: transformers
+tags:
+- MLLM
+---
+
+## Introduction
+Ovis is a novel Multimodal Large Language Model (MLLM) architecture, designed to structurally align visual and textual embeddings. For a comprehensive introduction, please refer to [Ovis paper](https://arxiv.org/abs/2405.20797) and [Ovis GitHub](https://github.com/AIDC-AI/Ovis).
+
+## Model
+Ovis can be instantiated with popular LLMs (e.g., Qwen, Llama3). We provide the following pretrained Ovis MLLMs:
+
+|                |                                               Ovis-Clip-Qwen1.5-7B |                                               Ovis-Clip-Llama3-8B |                                               Ovis-Clip-Qwen1.5-14B |
+|:---------------|-------------------------------------------------------------------:|------------------------------------------------------------------:|--------------------------------------------------------------------:|
+| ViT            |                                                               Clip |                                                              Clip |                                                                Clip |
+| LLM            |                                                    Qwen1.5-7B-Chat |                                                Llama3-8B-Instruct |                                                    Qwen1.5-14B-Chat |
+| Download       | [Huggingface](https://huggingface.co/AIDC-AI/Ovis-Clip-Qwen1_5-7B) | [Huggingface](https://huggingface.co/AIDC-AI/Ovis-Clip-Llama3-8B) | [Huggingface](https://huggingface.co/AIDC-AI/Ovis-Clip-Qwen1_5-14B) |
+| MMStar         |                                                               44.3 |                                                              49.5 |                                                                48.5 |
+| MMB-EN         |                                                               75.1 |                                                              77.4 |                                                                78.4 |
+| MMB-CN         |                                                               70.2 |                                                              72.8 |                                                                76.6 |
+| MMMU-Val       |                                                               39.7 |                                                              44.7 |                                                                46.7 |
+| MMMU-Test      |                                                               37.7 |                                                              39.0 |                                                                40.7 |
+| MathVista-Mini |                                                               41.4 |                                                              40.8 |                                                                43.4 |
+| MME            |                                                               1882 |                                                              2009 |                                                                1961 |
+| HallusionBench |                                                               56.4 |                                                              61.1 |                                                                57.6 |
+| RealWorldQA    |                                                               60.0 |                                                              57.9 |                                                                62.7 |
+
+## Usage
+Below is a code snippet to run Ovis with multimodal inputs. For additional usage instructions, including inference wrapper and Gradio UI, please refer to [Ovis GitHub](https://github.com/AIDC-AI/Ovis).
+```bash
+pip install torch==2.1.0 transformers==4.41.1 deepspeed==0.14.0 pillow==10.3.0
+```
+```python
+import torch
+from PIL import Image
+from transformers import AutoModelForCausalLM
+
+# load model
+model = AutoModelForCausalLM.from_pretrained("AIDC-AI/Ovis-Clip-Qwen1_5-7B",
+                                             torch_dtype=torch.bfloat16,
+                                             multimodal_max_length=8192,
+                                             trust_remote_code=True).cuda()
+text_tokenizer = model.get_text_tokenizer()
+visual_tokenizer = model.get_visual_tokenizer()
+conversation_formatter = model.get_conversation_formatter()
+
+# enter image path and prompt
+image_path = input("Enter image path: ")
+image = Image.open(image_path)
+text = input("Enter prompt: ")
+query = f'<image> {text}'
+prompt, input_ids = conversation_formatter.format_query(query)
+input_ids = torch.unsqueeze(input_ids, dim=0).to(device=model.device)
+attention_mask = torch.ne(input_ids, text_tokenizer.pad_token_id).to(device=model.device)
+pixel_values = [visual_tokenizer.preprocess_image(image).to(
+    dtype=visual_tokenizer.dtype, device=visual_tokenizer.device)]
+
+# print model output
+with torch.inference_mode():
+    kwargs = dict(
+        pixel_values=pixel_values,
+        attention_mask=attention_mask,
+        do_sample=False,
+        top_p=None,
+        temperature=None,
+        top_k=None,
+        repetition_penalty=None,
+        max_new_tokens=512,
+        use_cache=True,
+        eos_token_id=text_tokenizer.eos_token_id,
+        pad_token_id=text_tokenizer.pad_token_id
+    )
+    output_ids = model.generate(input_ids, **kwargs)[0]
+    input_token_len = input_ids.shape[1]
+    output = text_tokenizer.decode(output_ids[input_token_len:], skip_special_tokens=True)
+    print(f'Output: {output}')
+```
+
+## Citation
+If you find Ovis useful, please cite the paper
+```
+@article{lu2024ovis,
+  title={Ovis: Structural Embedding Alignment for Multimodal Large Language Model}, 
+  author={Shiyin Lu and Yang Li and Qing-Guo Chen and Zhao Xu and Weihua Luo and Kaifu Zhang and Han-Jia Ye},
+  year={2024},
+  journal={arXiv:2405.20797}
+}
+```
+  
+## License
+The project is licensed under the Apache 2.0 License and is restricted to uses that comply with the license agreements of Qwen, Llama3, and Clip.

added_tokens.json

@@ -0,0 +1,5 @@
+{
+  "<|endoftext|>": 151643,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644
+}

base_visual_tokenizer.py

@@ -0,0 +1,138 @@
+from typing import Union, Optional
+
+import PIL.Image
+import torch
+from torch.nn.functional import softmax, gumbel_softmax
+from transformers import PretrainedConfig, PreTrainedModel, AutoImageProcessor, AutoModel, AutoConfig
+
+
+class BaseVisualTokenizerConfig(PretrainedConfig):
+    def __init__(self,
+                 vocab_size=16384,
+                 tokenize_function="softmax",
+                 tau=1.0,
+                 depths=None,
+                 use_indicators=False,
+                 drop_cls_token=False,
+                 backbone_config: Optional[Union[PretrainedConfig, dict]] = None,
+                 hidden_stride: int = 1,
+                 **kwargs):
+        super().__init__(**kwargs)
+        self.vocab_size = vocab_size
+        self.tokenize_function = tokenize_function
+        self.tau = tau
+        if isinstance(depths, str):
+            depths = [int(x) for x in depths.split('|')]
+        self.depths = depths
+        self.backbone_kwargs = {}
+        self.use_indicators = use_indicators
+        self.drop_cls_token = drop_cls_token
+        if backbone_config is not None:
+            assert isinstance(backbone_config, (PretrainedConfig, dict)), \
+                f"expect `backbone_config` to be instance of PretrainedConfig or dict, but got {type(backbone_config)} type"
+            if not isinstance(backbone_config, PretrainedConfig):
+                model_type = backbone_config['model_type']
+                backbone_config.pop('model_type')
+                backbone_config = AutoConfig.for_model(model_type, **backbone_config)
+        self.backbone_config = backbone_config
+        self.hidden_stride = hidden_stride
+
+
+class BaseVisualTokenizer(PreTrainedModel):
+    base_model_prefix = "backbone"
+    main_input_name = None
+    _image_processor_class = None
+    _image_processor_kwargs = {}
+    _backbone_class = None
+    _backbone_name_or_path = None
+
+    def __init__(self, config: BaseVisualTokenizerConfig, *inputs, **kwargs):
+        super().__init__(config, *inputs, **kwargs)
+        if kwargs.get('train_from_scratch'):
+            self.image_processor = self._image_processor_class.from_pretrained(self._backbone_name_or_path,
+                                                                               **self._image_processor_kwargs)
+            self.backbone = self._backbone_class.from_pretrained(self._backbone_name_or_path,
+                                                                 **self.config.backbone_kwargs)
+            self.config.backbone_config = self.backbone.config
+        else:
+            self.image_processor = AutoImageProcessor.from_pretrained(kwargs['image_processor_name_or_path'])
+            self.backbone = AutoModel.from_config(self.config.backbone_config)
+        self.head = None
+
+        assert all((self.image_processor.do_resize,
+                    not getattr(self.image_processor, 'do_center_crop', False),
+                    self.image_processor.do_rescale,
+                    self.image_processor.do_normalize
+                    )), f"image_processor `{self.image_processor}` is not supported currently"
+
+    def get_backbone(self):
+        return self.backbone
+
+    def get_monitor_tensors(self):
+        raise NotImplementedError
+
+    def get_image_processor(self):
+        return self.image_processor
+
+    def get_head(self):
+        return self.head
+
+    def get_image_size(self):
+        raise NotImplementedError
+
+    def preprocess_image(self, image: PIL.Image.Image, convert_to_rgb=True):
+        if convert_to_rgb and image.mode != 'RGB':
+            image = image.convert('RGB')
+
+        # first resize and preprocess
+        sides = self.get_image_size()
+        if sides[0] != sides[1]:
+            raise ValueError('get_image_size() returns non-square size')
+        side = sides[0]
+
+        width, height = image.size
+        if width == height:
+            new_width = new_height = side
+        elif width > height:
+            new_width = side
+            new_height = int(height / width * new_width)
+        else:
+            new_height = side
+            new_width = int(width / height * new_height)
+        new_size = dict(height=new_height, width=new_width)
+        pixel_values = self.image_processor.preprocess(image, size=new_size, return_tensors='pt')['pixel_values']
+
+        # then pad to square
+        square_values = torch.zeros([1, 3, side, side], dtype=pixel_values.dtype, device=pixel_values.device)
+        new_height, new_width = pixel_values.shape[2:]
+        if new_height == new_width:
+            square_values[:, :, :, :] = pixel_values
+        elif new_height > new_width:
+            from_index = (side - new_width) // 2
+            square_values[:, :, :, from_index:from_index + new_width] = pixel_values
+        else:
+            from_index = (side - new_height) // 2
+            square_values[:, :, from_index:from_index + new_height, :] = pixel_values
+
+        return square_values
+
+    def get_layer_norm(self):
+        return self.layer_norm
+
+    def tokenize(self, logits):
+        def st_argmax(y_soft, dim):  # straight-through softmax
+            index = y_soft.max(dim, keepdim=True)[1]
+            y_hard = torch.zeros_like(y_soft, memory_format=torch.legacy_contiguous_format).scatter_(dim, index, 1.0)
+            ret = y_hard - y_soft.detach() + y_soft
+            return ret
+
+        if self.config.tokenize_function == 'softmax':
+            tokens = softmax(logits, dim=-1)
+        elif self.config.tokenize_function == 'gumbel_argmax':
+            tokens = gumbel_softmax(logits, tau=self.config.tau, hard=True)
+        elif self.config.tokenize_function == 'st_argmax':
+            tokens = st_argmax(logits, dim=-1)
+        else:
+            raise ValueError(
+                f'Invalid `max_type`, expected softmax or gumbel_argmax or st_argmax, but got {self.config.tokenize_function}')
+        return tokens

clip_visual_tokenizer.py

@@ -0,0 +1,120 @@
+from datetime import datetime
+from typing import Dict
+
+import deepspeed
+import torch
+from torch import Tensor
+from transformers import AutoConfig, AutoModel
+from transformers import CLIPVisionModel, CLIPImageProcessor
+from transformers.integrations import is_deepspeed_zero3_enabled
+
+from .utils import BEGIN_LINE, END_LINE, rank0_print
+from .base_visual_tokenizer import BaseVisualTokenizerConfig, BaseVisualTokenizer
+
+MODEL_TYPE = "clip_visual_tokenizer"
+
+
+class ClipVisualTokenizerConfig(BaseVisualTokenizerConfig):
+    model_type = MODEL_TYPE
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        if self.depths:
+            assert len(self.depths) == 1
+            self.backbone_kwargs['num_hidden_layers'] = self.depths[0]
+
+
+class ClipVisualTokenizer(BaseVisualTokenizer):
+    config_class = ClipVisualTokenizerConfig
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["CLIPEncoderLayer"]
+    _image_processor_class = CLIPImageProcessor
+    _image_processor_kwargs = dict(do_center_crop=False)
+    _backbone_class = CLIPVisionModel
+    _backbone_name_or_path = "openai/clip-vit-large-patch14-336"
+
+    def __init__(self, config: ClipVisualTokenizerConfig = None, *inputs, **kwargs):
+        super().__init__(config, *inputs, **kwargs)
+        head_dim = self.config.vocab_size
+        if self.config.use_indicators:
+            head_dim -= 2  # reserved for two image indicator tokens
+        self.head = torch.nn.Sequential(
+            torch.nn.Linear(self.backbone.config.hidden_size, head_dim, bias=False),
+            torch.nn.LayerNorm(head_dim)
+        )
+
+    def re_init_layers(self, re_init_layer_begin):
+        layer_dict = self.get_re_init_layer_dict(re_init_layer_begin)
+        for name, layer in layer_dict.items():
+            rank0_print(BEGIN_LINE)
+            rank0_print(f'[{datetime.now()}] Before layer re-initialization of {name}: ')
+            for k, v in layer.named_parameters():
+                with deepspeed.zero.GatheredParameters([v]):
+                    rank0_print(f'{k}: {v}')
+            with deepspeed.zero.GatheredParameters(list(layer.parameters(recurse=True)), modifier_rank=0):
+                if not is_deepspeed_zero3_enabled() or deepspeed.comm.get_rank() == 0:
+                    layer.apply(self.backbone._init_weights)
+            rank0_print(f'[{datetime.now()}] After layer re-initialization of {name}:')
+            for k, v in layer.named_parameters():
+                with deepspeed.zero.GatheredParameters([v]):
+                    rank0_print(f'{k}: {v}')
+            rank0_print(END_LINE)
+
+    def get_re_init_layer_dict(self, re_init_layer_begin: int) -> Dict[str, torch.nn.Module]:
+        assert re_init_layer_begin >= 0, "negative index is prohibited"
+        layer_dict = dict()
+        for i in range(re_init_layer_begin, self.backbone.config.num_hidden_layers):
+            layer_dict[f'backbone.vision_model.encoder.layers.{i}'] = self.backbone.vision_model.encoder.layers[i]
+        return layer_dict
+
+    def get_monitor_tensors(self):
+        return dict(
+            backbone_bottom=self.backbone.vision_model.encoder.layers[0].self_attn.k_proj.weight,
+            backbone_top=self.backbone.vision_model.encoder.layers[-1].self_attn.out_proj.weight,
+            head=self.head[0].weight
+        )
+
+    def get_image_size(self):
+        height = self.image_processor.crop_size["height"]
+        width = self.image_processor.crop_size["width"]
+        return height, width
+
+    def forward(self, pixel_values) -> Tensor:  # [BatchSize, ImageShape] -> [BatchSize, #Token, VocabSize]
+        output = self.backbone(
+            pixel_values, output_hidden_states=True, return_dict=True)
+        features = output.last_hidden_state
+        if self.config.drop_cls_token:
+            features = features[:, 1:, :]
+        logits = self.head(features)
+        tokens = self.tokenize(logits)
+        if self.config.use_indicators:
+            # tokens' shape is [BatchSize, #Token, VocabSize-2], so padding with [BatchSize, #Token, 2], after
+            # which, tokens' shape should become [BatchSize, #Token, VocabSize]
+            batch_size, token_len, _ = tokens.shape
+            padding_tensor = torch.zeros(size=(batch_size, token_len, 2),
+                                         dtype=tokens.dtype,
+                                         device=tokens.device,
+                                         layout=tokens.layout,
+                                         requires_grad=False)
+            tokens = torch.cat((tokens, padding_tensor), dim=2)
+
+            # adding indicator tokens, after which tokens' shape should become [BatchSize, 1+#Token+1, VocabSize]
+            begin_indicator = torch.zeros(size=(batch_size, 1),
+                                          dtype=torch.long,
+                                          device=tokens.device,
+                                          requires_grad=False) + self.config.vocab_size - 2
+            begin_indicator_token = torch.nn.functional.one_hot(begin_indicator,
+                                                                num_classes=self.config.vocab_size).to(
+                dtype=tokens.dtype)
+            end_indicator = torch.zeros(size=(batch_size, 1),
+                                        dtype=torch.long,
+                                        device=tokens.device,
+                                        requires_grad=False) + self.config.vocab_size - 1
+            end_indicator_token = torch.nn.functional.one_hot(end_indicator,
+                                                              num_classes=self.config.vocab_size).to(dtype=tokens.dtype)
+            tokens = torch.cat((begin_indicator_token, tokens, end_indicator_token), dim=1)
+        return tokens
+
+
+AutoConfig.register(MODEL_TYPE, ClipVisualTokenizerConfig)
+AutoModel.register(ClipVisualTokenizerConfig, ClipVisualTokenizer)

config.json

@@ -0,0 +1,245 @@
+{
+  "architectures": [
+    "Ovis"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_ovis.OvisConfig",
+    "AutoModelForCausalLM": "modeling_ovis.Ovis"
+  },
+  "conversation_formatter_class": "QwenConversationFormatter",
+  "hidden_size": 4096,
+  "llm_config": {
+    "_name_or_path": "Qwen/Qwen1.5-7B-Chat",
+    "add_cross_attention": false,
+    "architectures": [
+      "Qwen2ForCausalLM"
+    ],
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": 151643,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": 151645,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_act": "silu",
+    "hidden_size": 4096,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 11008,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "max_position_embeddings": 32768,
+    "max_window_layers": 28,
+    "min_length": 0,
+    "model_type": "qwen2",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 32,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 32,
+    "num_key_value_heads": 32,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "rms_norm_eps": 1e-06,
+    "rope_theta": 1000000.0,
+    "sep_token_id": null,
+    "sliding_window": 32768,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": false,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": "bfloat16",
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_cache": true,
+    "use_sliding_window": false,
+    "vocab_size": 151936
+  },
+  "model_type": "ovis",
+  "multimodal_max_length": 2048,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.41.1",
+  "use_cache": true,
+  "visual_tokenizer_config": {
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": null,
+    "backbone_config": {
+      "_name_or_path": "openai/clip-vit-large-patch14-336",
+      "add_cross_attention": false,
+      "architectures": null,
+      "attention_dropout": 0.0,
+      "bad_words_ids": null,
+      "begin_suppress_tokens": null,
+      "bos_token_id": null,
+      "chunk_size_feed_forward": 0,
+      "cross_attention_hidden_size": null,
+      "decoder_start_token_id": null,
+      "diversity_penalty": 0.0,
+      "do_sample": false,
+      "dropout": 0.0,
+      "early_stopping": false,
+      "encoder_no_repeat_ngram_size": 0,
+      "eos_token_id": null,
+      "exponential_decay_length_penalty": null,
+      "finetuning_task": null,
+      "forced_bos_token_id": null,
+      "forced_eos_token_id": null,
+      "hidden_act": "quick_gelu",
+      "hidden_size": 1024,
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "image_size": 336,
+      "initializer_factor": 1.0,
+      "initializer_range": 0.02,
+      "intermediate_size": 4096,
+      "is_decoder": false,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "layer_norm_eps": 1e-05,
+      "length_penalty": 1.0,
+      "max_length": 20,
+      "min_length": 0,
+      "model_type": "clip_vision_model",
+      "no_repeat_ngram_size": 0,
+      "num_attention_heads": 16,
+      "num_beam_groups": 1,
+      "num_beams": 1,
+      "num_channels": 3,
+      "num_hidden_layers": 24,
+      "num_return_sequences": 1,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "output_scores": false,
+      "pad_token_id": null,
+      "patch_size": 14,
+      "prefix": null,
+      "problem_type": null,
+      "projection_dim": 768,
+      "pruned_heads": {},
+      "remove_invalid_values": false,
+      "repetition_penalty": 1.0,
+      "return_dict": true,
+      "return_dict_in_generate": false,
+      "sep_token_id": null,
+      "suppress_tokens": null,
+      "task_specific_params": null,
+      "temperature": 1.0,
+      "tf_legacy_loss": false,
+      "tie_encoder_decoder": false,
+      "tie_word_embeddings": true,
+      "tokenizer_class": null,
+      "top_k": 50,
+      "top_p": 1.0,
+      "torch_dtype": null,
+      "torchscript": false,
+      "typical_p": 1.0,
+      "use_bfloat16": false
+    },
+    "backbone_kwargs": {},
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "depths": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "drop_cls_token": true,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_stride": 1,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "clip_visual_tokenizer",
+    "no_repeat_ngram_size": 0,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "tau": 1.0,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenize_function": "softmax",
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_indicators": true,
+    "vocab_size": 131072
+  }
+}

configuration_ovis.py

@@ -0,0 +1,35 @@
+from typing import Union, Optional
+
+from transformers import PretrainedConfig, AutoConfig
+from .visual_tokenizer import ClipVisualTokenizerConfig
+
+class OvisConfig(PretrainedConfig):
+    model_type = "ovis"
+
+    def __init__(self,
+                 llm_config: Optional[Union[PretrainedConfig, dict]] = None,
+                 visual_tokenizer_config: Optional[Union[PretrainedConfig, dict]] = None,
+                 multimodal_max_length=2048,
+                 hidden_size=None,
+                 conversation_formatter_class=None,
+                 **kwargs):
+        super().__init__(**kwargs)
+        if llm_config is not None:
+            assert isinstance(llm_config, (PretrainedConfig, dict)), \
+                f"expect `llm_config` to be instance of PretrainedConfig or dict, but got {type(llm_config)} type"
+            if not isinstance(llm_config, PretrainedConfig):
+                model_type = llm_config['model_type']
+                llm_config.pop('model_type')
+                llm_config = AutoConfig.for_model(model_type, **llm_config)
+        self.llm_config = llm_config
+        if visual_tokenizer_config is not None:
+            assert isinstance(visual_tokenizer_config, (PretrainedConfig, dict)), \
+                f"expect `visual_tokenizer_config` to be instance of PretrainedConfig or dict, but got {type(visual_tokenizer_config)} type"
+            if not isinstance(visual_tokenizer_config, PretrainedConfig):
+                model_type = visual_tokenizer_config['model_type']
+                visual_tokenizer_config.pop('model_type')
+                visual_tokenizer_config = AutoConfig.for_model(model_type, **visual_tokenizer_config)
+        self.visual_tokenizer_config = visual_tokenizer_config
+        self.multimodal_max_length = multimodal_max_length
+        self.hidden_size = hidden_size
+        self.conversation_formatter_class = conversation_formatter_class

conversation_formatter.py

@@ -0,0 +1,190 @@
+from abc import ABC, abstractmethod
+from typing import List, Dict
+
+import torch
+
+from .utils import IMAGE_TOKEN_INDEX, IGNORE_INDEX, IMAGE_TOKEN
+
+
+class ConversationFormatter(ABC):
+    support_tokenizer_types = None
+
+    def __init__(self, tokenizer):
+        tokenizer_type = type(tokenizer).__name__
+        assert tokenizer_type in self.support_tokenizer_types, \
+            f'Invalid tokenizer type, expected one from `{self.support_tokenizer_types}`, but got `{tokenizer_type}`'
+
+    @abstractmethod
+    def format(self, conversations: List[Dict], generation_preface=None):
+        pass
+
+    @abstractmethod
+    def format_query(self, query, generation_preface=""):
+        pass
+
+
+class QwenConversationFormatter(ConversationFormatter):
+    support_tokenizer_types = ['QWenTokenizer', 'Qwen2TokenizerFast']
+
+    def __init__(self, tokenizer):
+        super().__init__(tokenizer)
+        self.tokenizer = tokenizer
+        self.from2role = {
+            "system": "<|im_start|>system\n",
+            "human": "<|im_start|>user\n",
+            "gpt": "<|im_start|>assistant\n",
+        }
+        self.gpt_token_num = None
+        self.im_end = "<|im_end|>"
+        self.image_symbol = IMAGE_TOKEN
+        self.image_token_index = IMAGE_TOKEN_INDEX
+        self.ignore_index = IGNORE_INDEX
+        self.default_system_prompt = "You are a helpful assistant."
+
+    def _tokenize_with_image_symbol(self, text):
+        text_chunks = [self.tokenizer(chunk, add_special_tokens=False).input_ids for chunk in
+                       text.split(self.image_symbol)]
+        token_ids = []
+        num_chuck = len(text_chunks)
+        for i, chunk in enumerate(text_chunks):
+            token_ids.extend(chunk)
+            if i < num_chuck - 1:
+                token_ids.append(self.image_token_index)
+        return token_ids
+
+    def format(self, conversations: List[Dict], generation_preface=None):
+        if self.gpt_token_num is None:
+            self.gpt_token_num = len(self.tokenizer(self.from2role["gpt"], add_special_tokens=False).input_ids)
+
+        if conversations[0]["from"] != "system":
+            conversations.insert(0, {
+                "from": "system",
+                "value": self.default_system_prompt
+            })
+
+        if generation_preface is not None:
+            conversations.append({
+                "from": "gpt",
+                "value": generation_preface
+            })
+
+        prompt = ""
+        input_ids = []
+        labels = []
+        num_conversation = len(conversations)
+        for i, conversation in enumerate(conversations):
+            frm = conversation["from"]
+            role = self.from2role[frm]
+            message = conversation["value"]
+            text = role + message
+            if i < num_conversation - 1 or generation_preface is None:
+                text += self.im_end
+            if i < num_conversation - 1:
+                text += '\n'
+            prompt += text
+            token_ids = self._tokenize_with_image_symbol(text)
+            input_ids.extend(token_ids)
+            label_ids = [self.ignore_index] * len(token_ids)
+            if frm == "gpt":
+                label_ids[self.gpt_token_num:] = token_ids[self.gpt_token_num:]
+            labels.extend(label_ids)
+
+        assert self._tokenize_with_image_symbol(prompt) == input_ids
+        assert len(input_ids) == len(labels)
+        input_ids = torch.tensor(input_ids, dtype=torch.long)
+        labels = torch.tensor(labels, dtype=torch.long)
+
+        return prompt, input_ids, labels
+
+    def format_query(self, query, generation_preface=""):
+        prompt, input_ids, _ = self.format([{
+            "from": "human",
+            "value": query
+        }], generation_preface=generation_preface)
+
+        return prompt, input_ids
+
+
+class Llama3ConversationFormatter(ConversationFormatter):
+    support_tokenizer_types = ['PreTrainedTokenizerFast']
+
+    def __init__(self, tokenizer):
+        super().__init__(tokenizer)
+        self.tokenizer = tokenizer
+        self.from2role = {
+            "system": "<|start_header_id|>system<|end_header_id|>\n\n",
+            "human": "<|start_header_id|>user<|end_header_id|>\n\n",
+            "gpt": "<|start_header_id|>assistant<|end_header_id|>\n\n",
+        }
+        self.gpt_token_num = None
+        self.im_end = "<|eot_id|>"
+        self.image_symbol = IMAGE_TOKEN
+        self.image_token_index = IMAGE_TOKEN_INDEX
+        self.ignore_index = IGNORE_INDEX
+        self.default_system_prompt = "You are a helpful and honest multimodal assistant."
+        self.bos_token = "<|begin_of_text|>"
+        self.bos_token_ids = None
+
+    def _tokenize_with_image_symbol(self, text):
+        text_chunks = [self.tokenizer(chunk, add_special_tokens=False).input_ids for chunk in
+                       text.split(self.image_symbol)]
+        token_ids = []
+        num_chuck = len(text_chunks)
+        for i, chunk in enumerate(text_chunks):
+            token_ids.extend(chunk)
+            if i < num_chuck - 1:
+                token_ids.append(self.image_token_index)
+        return token_ids
+
+    def format(self, conversations: List[Dict], generation_preface=None):
+        if self.gpt_token_num is None:
+            self.gpt_token_num = len(self.tokenizer(self.from2role["gpt"], add_special_tokens=False).input_ids)
+
+        if self.bos_token_ids is None:
+            self.bos_token_ids = self.tokenizer(self.bos_token, add_special_tokens=False).input_ids
+
+        if conversations[0]["from"] != "system":
+            conversations.insert(0, {
+                "from": "system",
+                "value": self.default_system_prompt
+            })
+
+        if generation_preface is not None:
+            conversations.append({
+                "from": "gpt",
+                "value": generation_preface
+            })
+
+        prompt = "" + self.bos_token
+        input_ids = [] + self.bos_token_ids
+        labels = [] + [IGNORE_INDEX] * len(input_ids)
+        num_conversation = len(conversations)
+        for i, conversation in enumerate(conversations):
+            frm = conversation["from"]
+            role = self.from2role[frm]
+            message = conversation["value"].strip()
+            text = role + message
+            if i < num_conversation - 1 or generation_preface is None:
+                text += self.im_end
+            prompt += text
+            token_ids = self._tokenize_with_image_symbol(text)
+            input_ids.extend(token_ids)
+            label_ids = [self.ignore_index] * len(token_ids)
+            if frm == "gpt":
+                label_ids[self.gpt_token_num:] = token_ids[self.gpt_token_num:]
+            labels.extend(label_ids)
+
+        assert self._tokenize_with_image_symbol(prompt) == input_ids
+        assert len(input_ids) == len(labels)
+        input_ids = torch.tensor(input_ids, dtype=torch.long)
+        labels = torch.tensor(labels, dtype=torch.long)
+
+        return prompt, input_ids, labels
+
+    def format_query(self, query, generation_preface=""):
+        prompt, input_ids, _ = self.format([{
+            "from": "human",
+            "value": query
+        }], generation_preface=generation_preface)
+
+        return prompt, input_ids

generation_config.json

@@ -0,0 +1,15 @@
+{
+  "bos_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "multimodal_max_length": 2048,
+  "pad_token_id": 151643,
+  "repetition_penalty": 1.05,
+  "temperature": 0.7,
+  "top_k": 20,
+  "top_p": 0.8,
+  "transformers_version": "4.41.1"
+}

modeling_ovis.py

@@ -0,0 +1,289 @@
+import os
+from datetime import datetime
+from importlib import import_module
+from typing import List, Union, Callable, Optional
+
+import deepspeed
+import torch
+from torch import Tensor, LongTensor, IntTensor
+from torch.nn import init
+from transformers import PreTrainedModel, AutoConfig, AutoModel, AutoTokenizer, AutoModelForCausalLM
+from transformers.integrations.deepspeed import is_deepspeed_zero3_enabled, deepspeed_config
+
+from .visual_tokenizer import ClipVisualTokenizer
+from .configuration_ovis import OvisConfig
+from .conversation_formatter import ConversationFormatter
+from .utils import IGNORE_INDEX, IMAGE_TOKEN_INDEX, BEGIN_LINE, END_LINE, rank0_print
+
+class VisualEmbedding(torch.nn.Embedding):
+    def forward(self, input: Tensor) -> Tensor:
+        if any((isinstance(input, LongTensor), isinstance(input, IntTensor))):
+            return super().forward(input)
+        return torch.matmul(input, self.weight)
+
+    def reset_parameters(self, mean=0., std=1.) -> None:
+        init.normal_(self.weight, mean=mean, std=std)
+        self._fill_padding_idx_with_zero()
+
+
+class OvisPreTrainedModel(PreTrainedModel):
+    config_class = OvisConfig
+    base_model_prefix = "ovis"
+
+
+class Ovis(OvisPreTrainedModel):
+
+    def __init__(self, config: OvisConfig, *inputs, **kwargs):
+        super().__init__(config, *inputs, **kwargs)
+        if kwargs.get('train_from_scratch'):
+            self.llm = AutoModelForCausalLM.from_pretrained(kwargs['llm_name_or_path'], token=None)  # add token for gated model
+            self.generation_config = self.llm.generation_config
+            self.config.llm_config = self.llm.config
+            self.config.hidden_size = self.llm.config.hidden_size  # for deepspeed auto configuration
+            self.text_tokenizer = AutoTokenizer.from_pretrained(kwargs['llm_name_or_path'], token=None)  # add token for gated model
+            if self.text_tokenizer.pad_token_id is None and kwargs.get('pad_token_id') is not None:
+                self.text_tokenizer.pad_token_id = kwargs['pad_token_id']
+            if kwargs.get('visual_tokenizer_pretrained_path'):
+                self.visual_tokenizer = AutoModel.from_pretrained(kwargs['visual_tokenizer_pretrained_path'],
+                                                                  image_processor_name_or_path=kwargs[
+                                                                      'visual_tokenizer_pretrained_path'])
+            else:
+                self.visual_tokenizer = AutoModel.from_config(self.config.visual_tokenizer_config,
+                                                              train_from_scratch=True)
+            self.config.visual_tokenizer_config = self.visual_tokenizer.config
+        else:
+            self.llm = AutoModelForCausalLM.from_config(self.config.llm_config)
+            assert self.config.hidden_size == self.llm.config.hidden_size, "hidden size mismatch"
+            self.text_tokenizer = AutoTokenizer.from_pretrained(self.config.name_or_path)
+            self.visual_tokenizer = AutoModel.from_config(self.config.visual_tokenizer_config,
+                                                          image_processor_name_or_path=self.config.name_or_path)
+
+        # initialize vte
+        if is_deepspeed_zero3_enabled():
+            with deepspeed.zero.Init(config_dict_or_path=deepspeed_config()):
+                self.vte = VisualEmbedding(self.config.visual_tokenizer_config.vocab_size, self.config.hidden_size)
+        else:
+            self.visual_tokenizer.to(device=self.llm.device)
+            self.vte = VisualEmbedding(self.config.visual_tokenizer_config.vocab_size, self.config.hidden_size,
+                                       device=self.visual_tokenizer.device, dtype=self.visual_tokenizer.dtype)
+
+        def _merge_modules(modules_list: tuple):
+            merged_modules = []
+            for modules in modules_list:
+                merged_modules.extend(modules if modules else [])
+            return merged_modules
+
+        self._no_split_modules = _merge_modules((self.llm._no_split_modules, self.visual_tokenizer._no_split_modules))
+        self._skip_keys_device_placement = self.llm._skip_keys_device_placement
+        self._keep_in_fp32_modules = _merge_modules(
+            (self.llm._keep_in_fp32_modules, self.visual_tokenizer._keep_in_fp32_modules))
+        self.is_parallelizable = all((self.llm.is_parallelizable, self.visual_tokenizer.is_parallelizable))
+        self.supports_gradient_checkpointing = all(
+            (self.llm.supports_gradient_checkpointing, self.visual_tokenizer.supports_gradient_checkpointing))
+        self._supports_flash_attn_2 = all(
+            (self.llm._supports_flash_attn_2, self.visual_tokenizer._supports_flash_attn_2))
+        self._supports_sdpa = all((self.llm._supports_sdpa, self.visual_tokenizer._supports_sdpa))
+        self._supports_cache_class = all((self.llm._supports_cache_class, self.visual_tokenizer._supports_cache_class))
+
+    def get_text_tokenizer(self):
+        return self.text_tokenizer
+
+    def get_visual_tokenizer(self):
+        return self.visual_tokenizer
+
+    def re_init_vte(self, mean, std):
+        vte = self.get_vte()
+        rank0_print(BEGIN_LINE)
+        rank0_print(f'[{datetime.now()}] Before re-initialization of vte: ')
+        with deepspeed.zero.GatheredParameters([vte.weight]):
+            rank0_print(f'vte.weight: {vte.weight}')
+        with deepspeed.zero.GatheredParameters([vte.weight], modifier_rank=0):
+            if not is_deepspeed_zero3_enabled() or deepspeed.comm.get_rank() == 0:
+                vte.reset_parameters(mean, std)
+        rank0_print(f'[{datetime.now()}] After re-initialization of vte:')
+        with deepspeed.zero.GatheredParameters([vte.weight]):
+            rank0_print(f'vte.weight: {vte.weight}')
+        rank0_print(END_LINE)
+
+    def get_monitor_tensors(self):
+        monitor_tensors = dict(
+            wte=self.get_wte().weight,
+            lm_head=self.get_lm_head().weight,
+            vte=self.get_vte().weight
+        )
+        monitor_tensors.update(
+            {f'visual_tokenizer_{k}': v for k, v in self.get_visual_tokenizer().get_monitor_tensors().items()})
+        return monitor_tensors
+
+    def get_lm_head(self):
+        return self.get_llm().get_output_embeddings()
+
+    def get_llm(self):
+        return self.llm
+
+    def get_vte(self):
+        return self.vte
+
+    def get_wte(self):
+        return self.llm.get_input_embeddings()
+
+    def get_conversation_formatter(self) -> ConversationFormatter:
+        if getattr(self, 'conversation_formatter', None) is None:
+            self.conversation_formatter = getattr(import_module(".conversation_formatter", __package__),
+                                                  self.config.conversation_formatter_class)(self.text_tokenizer)
+        return self.conversation_formatter
+
+    def forward(self, input_ids: torch.Tensor, attention_mask: torch.Tensor, pixel_values: List[Optional[torch.Tensor]],
+                labels: Optional[torch.Tensor] = None, **kwargs):
+        _, inputs_embeds, labels, attention_mask = self.merge_multimodal(
+            text_input_ids=input_ids,
+            text_attention_masks=attention_mask,
+            text_labels=labels,
+            pixel_values=pixel_values,
+            with_kv_cache=kwargs.get('past_key_values') is not None)
+        return self.llm(inputs_embeds=inputs_embeds, labels=labels, attention_mask=attention_mask, **kwargs)
+
+    def merge_multimodal(self, text_input_ids: torch.Tensor, text_attention_masks: torch.Tensor,
+                         text_labels: Optional[torch.Tensor], pixel_values: List[Optional[torch.Tensor]],
+                         with_kv_cache: bool = False):
+        if with_kv_cache:
+            return None, self.get_wte()(text_input_ids), text_labels, text_attention_masks
+        if self.training:
+            # When training, to be compatible with deepspeed zero, each sample has to include pixel_value tensor.
+            # For text-only sample, one can simply use a full zero tensor as pixel_value, which will be ignored
+            # (see below in this function); so, the gradient will not be affected.
+            num_images = [x.shape[0] for x in pixel_values]
+            visual_tokens = self.visual_tokenizer(torch.cat([x for x in pixel_values], dim=0))
+            visual_embeds = torch.split(self.get_vte()(visual_tokens).to(dtype=self.dtype),
+                                        split_size_or_sections=num_images, dim=0)
+            visual_input_ids = torch.split(torch.argmax(visual_tokens, dim=-1),
+                                           split_size_or_sections=num_images, dim=0)
+            visual_labels = [torch.full(x.shape, IGNORE_INDEX, dtype=torch.long) for x in visual_input_ids]
+        else:
+            # When inference, sample can include only text with `None` pixel_value
+            num_images = [x.shape[0] if x is not None else 0 for x in pixel_values]
+            if sum(num_images) > 0:
+                visual_tokens = self.visual_tokenizer(torch.cat([x for x in pixel_values if x is not None], dim=0))
+                visual_embeds = torch.split(self.get_vte()(visual_tokens).to(dtype=self.dtype),
+                                            split_size_or_sections=num_images, dim=0)
+                visual_input_ids = torch.split(torch.argmax(visual_tokens, dim=-1),
+                                               split_size_or_sections=num_images, dim=0)
+                visual_labels = [torch.full(x.shape, IGNORE_INDEX, dtype=torch.long) for x in visual_input_ids]
+            else:
+                # just placeholders
+                visual_embeds = [None] * len(num_images)
+                visual_input_ids = [None] * len(num_images)
+                visual_labels = [None] * len(num_images)
+            # just placeholders
+            text_labels = torch.full(text_input_ids.shape, IGNORE_INDEX, dtype=torch.long, device=text_input_ids.device)
+
+        input_embeds = []
+        attention_masks = []
+        labels = []
+        for text_input_id, text_label, text_attention_mask, visual_embed, visual_input_id, visual_label in zip(
+                text_input_ids, text_labels, text_attention_masks, visual_embeds, visual_input_ids, visual_labels
+        ):
+            image_token_mask = torch.eq(text_input_id, IMAGE_TOKEN_INDEX)
+            text_embed = self.get_wte()(torch.masked_fill(text_input_id, image_token_mask, 0))
+            image_token_positions = torch.where(image_token_mask)[0].tolist()
+            if len(image_token_positions) > 0:
+                input_embed_parts = []
+                attention_mask_parts = []
+                label_parts = []
+                prev_image_token_position = -1
+                for index, image_token_position in enumerate(image_token_positions):
+                    input_embed_parts.append(
+                        text_embed[prev_image_token_position + 1:image_token_position, :])
+                    label_parts.append(
+                        text_label[prev_image_token_position + 1:image_token_position])
+                    attention_mask_parts.append(
+                        text_attention_mask[prev_image_token_position + 1:image_token_position])
+                    input_embed_parts.append(visual_embed[index])
+                    attention_mask_parts.append(
+                        torch.ones_like(visual_label[index], device=text_label.device, dtype=torch.bool))
+                    label_parts.append(visual_label[index].to(device=text_label.device))
+                    prev_image_token_position = image_token_position
+                if prev_image_token_position + 1 < text_input_id.shape[0]:
+                    input_embed_parts.append(
+                        text_embed[prev_image_token_position + 1:, :])
+                    attention_mask_parts.append(
+                        text_attention_mask[prev_image_token_position + 1:])
+                    label_parts.append(
+                        text_label[prev_image_token_position + 1:])
+                input_embed = torch.cat(input_embed_parts, dim=0)
+                attention_mask = torch.cat(attention_mask_parts, dim=0)
+                label = torch.cat(label_parts, dim=0)
+            else:
+                input_embed = text_embed
+                attention_mask = text_attention_mask
+                label = text_label
+                if self.training:
+                    # Make visual_embed involved in the backward graph, to be compatible with deepspeed zero and ddp.
+                    input_embed += torch.sum(visual_embed * 0.0)
+            input_embeds.append(input_embed)
+            attention_masks.append(attention_mask)
+            labels.append(label)
+
+        batch_input_embeds = torch.nn.utils.rnn.pad_sequence(input_embeds, batch_first=True, padding_value=0.0)[:,
+                             :self.config.multimodal_max_length, :]
+        batch_attention_mask = torch.nn.utils.rnn.pad_sequence(attention_masks, batch_first=True, padding_value=False)[
+                               :,
+                               :self.config.multimodal_max_length]
+        batch_labels = torch.nn.utils.rnn.pad_sequence(labels, batch_first=True, padding_value=IGNORE_INDEX)[:,
+                       :self.config.multimodal_max_length]
+
+        return visual_input_ids, batch_input_embeds, batch_labels, batch_attention_mask
+
+    def save_pretrained(
+            self,
+            save_directory: Union[str, os.PathLike],
+            is_main_process: bool = True,
+            state_dict: Optional[dict] = None,
+            save_function: Callable = torch.save,
+            push_to_hub: bool = False,
+            max_shard_size: Union[int, str] = "5GB",
+            safe_serialization: bool = True,
+            variant: Optional[str] = None,
+            token: Optional[Union[str, bool]] = None,
+            save_peft_format: bool = True,
+            **kwargs,
+    ):
+        super().save_pretrained(save_directory,
+                                is_main_process=is_main_process,
+                                state_dict=state_dict,
+                                save_function=save_function,
+                                safe_serialization=safe_serialization)
+        self.get_text_tokenizer().save_pretrained(save_directory)
+        self.get_visual_tokenizer().get_image_processor().save_pretrained(save_directory)
+
+        # uncomment the following will additionally save a separate visual tokenizer
+        # visual_tokenizer_directory = os.path.join(save_directory, 'visual_tokenizer')
+        # self.get_visual_tokenizer().save_pretrained(visual_tokenizer_directory,
+        #                                             is_main_process=is_main_process,
+        #                                             state_dict=None,
+        #                                             save_function=save_function,
+        #                                             safe_serialization=safe_serialization)
+        # self.get_visual_tokenizer().get_image_processor().save_pretrained(visual_tokenizer_directory)
+
+    # TODO: support batch generation
+    def prepare_inputs_for_generation(
+            self, input_ids, pixel_values, attention_mask, past_key_values=None, inputs_embeds=None, **kwargs):
+        if past_key_values is not None:
+            input_ids = input_ids[:, -1:]
+            attention_mask = attention_mask[:, -1:]
+
+        # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
+        if inputs_embeds is not None and past_key_values is None:
+            model_inputs = {"inputs_embeds": inputs_embeds}
+        else:
+            model_inputs = {"input_ids": input_ids}
+
+        model_inputs.update(
+            {
+                "past_key_values": past_key_values,
+                "use_cache": kwargs.get("use_cache"),
+                "attention_mask": attention_mask,
+                "pixel_values": pixel_values
+            }
+        )
+        return model_inputs

preprocessor_config.json

@@ -0,0 +1,44 @@
+{
+  "_valid_processor_keys": [
+    "images",
+    "do_resize",
+    "size",
+    "resample",
+    "do_center_crop",
+    "crop_size",
+    "do_rescale",
+    "rescale_factor",
+    "do_normalize",
+    "image_mean",
+    "image_std",
+    "do_convert_rgb",
+    "return_tensors",
+    "data_format",
+    "input_data_format"
+  ],
+  "crop_size": {
+    "height": 336,
+    "width": 336
+  },
+  "do_center_crop": false,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "CLIPImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "shortest_edge": 336
+  }
+}

pytorch_model-00001-of-00004.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e687fcc19ec389ccd887de95af76503ec6036fc127d7dec62d5bd60fad1978d6
+size 4988508819

pytorch_model-00002-of-00004.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d5d35d3a58e406e898bc019a5bc9dd128c24021a66278d424ab5e63c5e025143
+size 4981277820

pytorch_model-00003-of-00004.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b6b23ff0b5ab2b06379f6f648164c88dc5d89df30427f81d779314d594095564
+size 4228335437

pytorch_model-00004-of-00004.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0082ecc301ec996fe1bd40d546964cc17394213fd790d216743eb8bc76caae26
+size 3194519786

pytorch_model.bin.index.json

@@ -0,0 +1,789 @@
+{
+  "metadata": {
+    "total_size": 17392361464
+  },
+  "weight_map": {
+    "llm.lm_head.weight": "pytorch_model-00004-of-00004.bin",
+    "llm.model.embed_tokens.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.0.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.0.mlp.down_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.0.mlp.gate_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.0.mlp.up_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.0.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.0.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.0.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.0.self_attn.o_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.0.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.0.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.0.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.0.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.1.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.1.mlp.down_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.1.mlp.gate_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.1.mlp.up_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.1.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.1.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.1.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.1.self_attn.o_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.1.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.1.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.1.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.1.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.10.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.10.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.10.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.10.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.10.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.10.self_attn.k_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.10.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.10.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.10.self_attn.q_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.10.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.10.self_attn.v_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.10.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.11.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.11.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.11.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.11.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.11.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.11.self_attn.k_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.11.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.11.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.11.self_attn.q_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.11.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.11.self_attn.v_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.11.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.12.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.12.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.12.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.12.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.12.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.12.self_attn.k_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.12.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.12.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.12.self_attn.q_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.12.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.12.self_attn.v_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.12.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.13.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.13.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.13.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.13.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.13.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.13.self_attn.k_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.13.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.13.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.13.self_attn.q_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.13.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.13.self_attn.v_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.13.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.14.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.14.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.14.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.14.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.14.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.14.self_attn.k_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.14.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.14.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.14.self_attn.q_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.14.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.14.self_attn.v_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.14.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.15.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.15.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.15.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.15.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.15.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.15.self_attn.k_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.15.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.15.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.15.self_attn.q_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.15.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.15.self_attn.v_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.15.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.16.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.16.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.16.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.16.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.16.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.16.self_attn.k_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.16.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.16.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.16.self_attn.q_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.16.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.16.self_attn.v_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.16.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.17.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.17.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.17.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.17.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.17.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.17.self_attn.k_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.17.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.17.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.17.self_attn.q_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.17.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.17.self_attn.v_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.17.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.18.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.18.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.18.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.18.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.18.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.18.self_attn.k_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.18.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.18.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.18.self_attn.q_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.18.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.18.self_attn.v_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.18.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.19.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.19.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.19.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.19.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.19.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.19.self_attn.k_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.19.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.19.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.19.self_attn.q_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.19.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.19.self_attn.v_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.19.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.2.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.2.mlp.down_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.2.mlp.gate_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.2.mlp.up_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.2.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.2.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.2.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.2.self_attn.o_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.2.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.2.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.2.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.2.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.20.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.20.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.20.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.20.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.20.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.20.self_attn.k_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.20.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.20.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.20.self_attn.q_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.20.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.20.self_attn.v_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.20.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.21.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.21.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.21.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.21.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.21.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.21.self_attn.k_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.21.self_attn.k_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.21.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.21.self_attn.q_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.21.self_attn.q_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.21.self_attn.v_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.21.self_attn.v_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.22.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.22.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.22.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.22.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.22.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.22.self_attn.k_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.22.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.22.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.22.self_attn.q_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.22.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.22.self_attn.v_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.22.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.23.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.23.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.23.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.23.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.23.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.23.self_attn.k_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.23.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.23.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.23.self_attn.q_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.23.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.23.self_attn.v_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.23.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.24.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.24.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.24.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.24.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.24.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.24.self_attn.k_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.24.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.24.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.24.self_attn.q_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.24.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.24.self_attn.v_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.24.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.25.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.25.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.25.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.25.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.25.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.25.self_attn.k_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.25.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.25.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.25.self_attn.q_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.25.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.25.self_attn.v_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.25.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.26.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.26.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.26.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.26.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.26.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.26.self_attn.k_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.26.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.26.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.26.self_attn.q_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.26.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.26.self_attn.v_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.26.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.27.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.27.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.27.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.27.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.27.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.27.self_attn.k_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.27.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.27.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.27.self_attn.q_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.27.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.27.self_attn.v_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.27.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.28.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.28.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.28.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.28.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.28.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.28.self_attn.k_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.28.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.28.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.28.self_attn.q_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.28.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.28.self_attn.v_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.28.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.29.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.29.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.29.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.29.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.29.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.29.self_attn.k_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.29.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.29.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.29.self_attn.q_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.29.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.29.self_attn.v_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.29.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.3.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.3.mlp.down_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.3.mlp.gate_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.3.mlp.up_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.3.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.3.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.3.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.3.self_attn.o_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.3.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.3.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.3.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.3.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.30.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.30.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.30.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.30.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.30.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.30.self_attn.k_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.30.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.30.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.30.self_attn.q_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.30.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.30.self_attn.v_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.30.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.31.input_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.31.mlp.down_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.31.mlp.gate_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.31.mlp.up_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.31.post_attention_layernorm.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.31.self_attn.k_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.31.self_attn.k_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.31.self_attn.o_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.31.self_attn.q_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.31.self_attn.q_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.31.self_attn.v_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.31.self_attn.v_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "llm.model.layers.4.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.4.mlp.down_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.4.mlp.gate_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.4.mlp.up_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.4.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.4.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.4.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.4.self_attn.o_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.4.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.4.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.4.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.4.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.5.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.5.mlp.down_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.5.mlp.gate_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.5.mlp.up_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.5.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.5.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.5.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.5.self_attn.o_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.5.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.5.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.5.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.5.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.6.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.6.mlp.down_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.6.mlp.gate_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.6.mlp.up_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.6.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.6.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.6.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.6.self_attn.o_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.6.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.6.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.6.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.6.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.7.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.7.mlp.down_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.7.mlp.gate_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.7.mlp.up_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.7.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.7.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.7.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.7.self_attn.o_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.7.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.7.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.7.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.7.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.8.input_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.8.mlp.down_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.8.mlp.gate_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.8.mlp.up_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.8.post_attention_layernorm.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.8.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.8.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.8.self_attn.o_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.8.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.8.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.8.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.8.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.9.input_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.9.mlp.down_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.9.mlp.gate_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.9.mlp.up_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.9.post_attention_layernorm.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.9.self_attn.k_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.9.self_attn.k_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.9.self_attn.o_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "llm.model.layers.9.self_attn.q_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.9.self_attn.q_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.9.self_attn.v_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "llm.model.layers.9.self_attn.v_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "llm.model.norm.weight": "pytorch_model-00003-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.embeddings.class_embedding": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.embeddings.patch_embedding.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.embeddings.position_embedding.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.0.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.1.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.10.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.11.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.12.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.13.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.14.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.15.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.16.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.17.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.18.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.19.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.2.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.20.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.21.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.22.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.23.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.3.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.4.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.5.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.6.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.7.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.8.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.layer_norm1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.layer_norm1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.layer_norm2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.layer_norm2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.mlp.fc1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.mlp.fc1.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.mlp.fc2.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.mlp.fc2.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.self_attn.k_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.self_attn.k_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.self_attn.out_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.self_attn.out_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.self_attn.q_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.self_attn.q_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.self_attn.v_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.encoder.layers.9.self_attn.v_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.post_layernorm.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.post_layernorm.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.pre_layrnorm.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.backbone.vision_model.pre_layrnorm.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.head.0.weight": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.head.1.bias": "pytorch_model-00004-of-00004.bin",
+    "visual_tokenizer.head.1.weight": "pytorch_model-00004-of-00004.bin",
+    "vte.weight": "pytorch_model-00004-of-00004.bin"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,20 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,43 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

utils.py

@@ -0,0 +1,35 @@
+import os
+from importlib import import_module
+
+# Model Constants
+IGNORE_INDEX = -100
+IMAGE_TOKEN_INDEX = -200
+IMAGE_TOKEN = "<image>"
+
+# Log & Print
+BEGIN_LINE = '========================************========================'
+END_LINE = '------------------------------------------------------------'
+
+
+def rank0_print(*args):
+    if int(os.getenv("LOCAL_PROCESS_RANK", os.getenv("LOCAL_RANK", 0))) == 0:
+        print(*args)
+
+
+def smart_unit(num):
+    if num / 1.0e9 >= 1:
+        return f'{num / 1.0e9:.2f}B'
+    else:
+        return f'{num / 1.0e6:.2f}M'
+
+
+def import_class_from_string(full_class_string):
+    # Split the path to get separate module and class names
+    module_path, _, class_name = full_class_string.rpartition('.')
+
+    # Import the module using the module path
+    module = import_module(module_path)
+
+    # Get the class from the imported module
+    cls = getattr(module, class_name)
+    return cls

visual_tokenizer.py

@@ -0,0 +1,254 @@
+from datetime import datetime
+from typing import Dict, Union, Optional
+
+import deepspeed
+import torch
+import PIL.Image
+from torch.nn.functional import softmax, gumbel_softmax
+from torch import Tensor
+from transformers import PretrainedConfig, PreTrainedModel, AutoImageProcessor, AutoConfig, AutoModel
+from transformers import CLIPVisionModel, CLIPImageProcessor
+from transformers.integrations import is_deepspeed_zero3_enabled
+
+from .utils import BEGIN_LINE, END_LINE, rank0_print
+
+MODEL_TYPE = "clip_visual_tokenizer"
+
+
+class BaseVisualTokenizerConfig(PretrainedConfig):
+    def __init__(self,
+                 vocab_size=16384,
+                 tokenize_function="softmax",
+                 tau=1.0,
+                 depths=None,
+                 use_indicators=False,
+                 drop_cls_token=False,
+                 backbone_config: Optional[Union[PretrainedConfig, dict]] = None,
+                 hidden_stride: int = 1,
+                 **kwargs):
+        super().__init__(**kwargs)
+        self.vocab_size = vocab_size
+        self.tokenize_function = tokenize_function
+        self.tau = tau
+        if isinstance(depths, str):
+            depths = [int(x) for x in depths.split('|')]
+        self.depths = depths
+        self.backbone_kwargs = {}
+        self.use_indicators = use_indicators
+        self.drop_cls_token = drop_cls_token
+        if backbone_config is not None:
+            assert isinstance(backbone_config, (PretrainedConfig, dict)), \
+                f"expect `backbone_config` to be instance of PretrainedConfig or dict, but got {type(backbone_config)} type"
+            if not isinstance(backbone_config, PretrainedConfig):
+                model_type = backbone_config['model_type']
+                backbone_config.pop('model_type')
+                backbone_config = AutoConfig.for_model(model_type, **backbone_config)
+        self.backbone_config = backbone_config
+        self.hidden_stride = hidden_stride
+
+
+class BaseVisualTokenizer(PreTrainedModel):
+    base_model_prefix = "backbone"
+    main_input_name = None
+    _image_processor_class = None
+    _image_processor_kwargs = {}
+    _backbone_class = None
+    _backbone_name_or_path = None
+
+    def __init__(self, config: BaseVisualTokenizerConfig, *inputs, **kwargs):
+        super().__init__(config, *inputs, **kwargs)
+        if kwargs.get('train_from_scratch'):
+            self.image_processor = self._image_processor_class.from_pretrained(self._backbone_name_or_path,
+                                                                               **self._image_processor_kwargs)
+            self.backbone = self._backbone_class.from_pretrained(self._backbone_name_or_path,
+                                                                 **self.config.backbone_kwargs)
+            self.config.backbone_config = self.backbone.config
+        else:
+            self.image_processor = AutoImageProcessor.from_pretrained(kwargs['image_processor_name_or_path'])
+            self.backbone = AutoModel.from_config(self.config.backbone_config)
+        self.head = None
+
+        assert all((self.image_processor.do_resize,
+                    not getattr(self.image_processor, 'do_center_crop', False),
+                    self.image_processor.do_rescale,
+                    self.image_processor.do_normalize
+                    )), f"image_processor `{self.image_processor}` is not supported currently"
+
+    def get_backbone(self):
+        return self.backbone
+
+    def get_monitor_tensors(self):
+        raise NotImplementedError
+
+    def get_image_processor(self):
+        return self.image_processor
+
+    def get_head(self):
+        return self.head
+
+    def get_image_size(self):
+        raise NotImplementedError
+
+    def preprocess_image(self, image: PIL.Image.Image, convert_to_rgb=True):
+        if convert_to_rgb and image.mode != 'RGB':
+            image = image.convert('RGB')
+
+        # first resize and preprocess
+        sides = self.get_image_size()
+        if sides[0] != sides[1]:
+            raise ValueError('get_image_size() returns non-square size')
+        side = sides[0]
+
+        width, height = image.size
+        if width == height:
+            new_width = new_height = side
+        elif width > height:
+            new_width = side
+            new_height = int(height / width * new_width)
+        else:
+            new_height = side
+            new_width = int(width / height * new_height)
+        new_size = dict(height=new_height, width=new_width)
+        pixel_values = self.image_processor.preprocess(image, size=new_size, return_tensors='pt')['pixel_values']
+
+        # then pad to square
+        square_values = torch.zeros([1, 3, side, side], dtype=pixel_values.dtype, device=pixel_values.device)
+        new_height, new_width = pixel_values.shape[2:]
+        if new_height == new_width:
+            square_values[:, :, :, :] = pixel_values
+        elif new_height > new_width:
+            from_index = (side - new_width) // 2
+            square_values[:, :, :, from_index:from_index + new_width] = pixel_values
+        else:
+            from_index = (side - new_height) // 2
+            square_values[:, :, from_index:from_index + new_height, :] = pixel_values
+
+        return square_values
+
+    def get_layer_norm(self):
+        return self.layer_norm
+
+    def tokenize(self, logits):
+        def st_argmax(y_soft, dim):  # straight-through softmax
+            index = y_soft.max(dim, keepdim=True)[1]
+            y_hard = torch.zeros_like(y_soft, memory_format=torch.legacy_contiguous_format).scatter_(dim, index, 1.0)
+            ret = y_hard - y_soft.detach() + y_soft
+            return ret
+
+        if self.config.tokenize_function == 'softmax':
+            tokens = softmax(logits, dim=-1)
+        elif self.config.tokenize_function == 'gumbel_argmax':
+            tokens = gumbel_softmax(logits, tau=self.config.tau, hard=True)
+        elif self.config.tokenize_function == 'st_argmax':
+            tokens = st_argmax(logits, dim=-1)
+        else:
+            raise ValueError(
+                f'Invalid `max_type`, expected softmax or gumbel_argmax or st_argmax, but got {self.config.tokenize_function}')
+        return tokens
+
+
+
+class ClipVisualTokenizerConfig(BaseVisualTokenizerConfig):
+    model_type = MODEL_TYPE
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        if self.depths:
+            assert len(self.depths) == 1
+            self.backbone_kwargs['num_hidden_layers'] = self.depths[0]
+
+
+class ClipVisualTokenizer(BaseVisualTokenizer):
+    config_class = ClipVisualTokenizerConfig
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["CLIPEncoderLayer"]
+    _image_processor_class = CLIPImageProcessor
+    _image_processor_kwargs = dict(do_center_crop=False)
+    _backbone_class = CLIPVisionModel
+    _backbone_name_or_path = "openai/clip-vit-large-patch14-336"
+
+    def __init__(self, config: ClipVisualTokenizerConfig = None, *inputs, **kwargs):
+        super().__init__(config, *inputs, **kwargs)
+        head_dim = self.config.vocab_size
+        if self.config.use_indicators:
+            head_dim -= 2  # reserved for two image indicator tokens
+        self.head = torch.nn.Sequential(
+            torch.nn.Linear(self.backbone.config.hidden_size, head_dim, bias=False),
+            torch.nn.LayerNorm(head_dim)
+        )
+
+    def re_init_layers(self, re_init_layer_begin):
+        layer_dict = self.get_re_init_layer_dict(re_init_layer_begin)
+        for name, layer in layer_dict.items():
+            rank0_print(BEGIN_LINE)
+            rank0_print(f'[{datetime.now()}] Before layer re-initialization of {name}: ')
+            for k, v in layer.named_parameters():
+                with deepspeed.zero.GatheredParameters([v]):
+                    rank0_print(f'{k}: {v}')
+            with deepspeed.zero.GatheredParameters(list(layer.parameters(recurse=True)), modifier_rank=0):
+                if not is_deepspeed_zero3_enabled() or deepspeed.comm.get_rank() == 0:
+                    layer.apply(self.backbone._init_weights)
+            rank0_print(f'[{datetime.now()}] After layer re-initialization of {name}:')
+            for k, v in layer.named_parameters():
+                with deepspeed.zero.GatheredParameters([v]):
+                    rank0_print(f'{k}: {v}')
+            rank0_print(END_LINE)
+
+    def get_re_init_layer_dict(self, re_init_layer_begin: int) -> Dict[str, torch.nn.Module]:
+        assert re_init_layer_begin >= 0, "negative index is prohibited"
+        layer_dict = dict()
+        for i in range(re_init_layer_begin, self.backbone.config.num_hidden_layers):
+            layer_dict[f'backbone.vision_model.encoder.layers.{i}'] = self.backbone.vision_model.encoder.layers[i]
+        return layer_dict
+
+    def get_monitor_tensors(self):
+        return dict(
+            backbone_bottom=self.backbone.vision_model.encoder.layers[0].self_attn.k_proj.weight,
+            backbone_top=self.backbone.vision_model.encoder.layers[-1].self_attn.out_proj.weight,
+            head=self.head[0].weight
+        )
+
+    def get_image_size(self):
+        height = self.image_processor.crop_size["height"]
+        width = self.image_processor.crop_size["width"]
+        return height, width
+
+    def forward(self, pixel_values) -> Tensor:  # [BatchSize, ImageShape] -> [BatchSize, #Token, VocabSize]
+        output = self.backbone(
+            pixel_values, output_hidden_states=True, return_dict=True)
+        features = output.last_hidden_state
+        if self.config.drop_cls_token:
+            features = features[:, 1:, :]
+        logits = self.head(features)
+        tokens = self.tokenize(logits)
+        if self.config.use_indicators:
+            # tokens' shape is [BatchSize, #Token, VocabSize-2], so padding with [BatchSize, #Token, 2], after
+            # which, tokens' shape should become [BatchSize, #Token, VocabSize]
+            batch_size, token_len, _ = tokens.shape
+            padding_tensor = torch.zeros(size=(batch_size, token_len, 2),
+                                         dtype=tokens.dtype,
+                                         device=tokens.device,
+                                         layout=tokens.layout,
+                                         requires_grad=False)
+            tokens = torch.cat((tokens, padding_tensor), dim=2)
+
+            # adding indicator tokens, after which tokens' shape should become [BatchSize, 1+#Token+1, VocabSize]
+            begin_indicator = torch.zeros(size=(batch_size, 1),
+                                          dtype=torch.long,
+                                          device=tokens.device,
+                                          requires_grad=False) + self.config.vocab_size - 2
+            begin_indicator_token = torch.nn.functional.one_hot(begin_indicator,
+                                                                num_classes=self.config.vocab_size).to(
+                dtype=tokens.dtype)
+            end_indicator = torch.zeros(size=(batch_size, 1),
+                                        dtype=torch.long,
+                                        device=tokens.device,
+                                        requires_grad=False) + self.config.vocab_size - 1
+            end_indicator_token = torch.nn.functional.one_hot(end_indicator,
+                                                              num_classes=self.config.vocab_size).to(dtype=tokens.dtype)
+            tokens = torch.cat((begin_indicator_token, tokens, end_indicator_token), dim=1)
+        return tokens
+
+
+AutoConfig.register(MODEL_TYPE, ClipVisualTokenizerConfig)
+AutoModel.register(ClipVisualTokenizerConfig, ClipVisualTokenizer)