Duplicate from vidore/colSmol-500M

Co-authored-by: Manuel Faysse <[email protected]>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,121 @@
+---
+license: mit
+library_name: colpali
+base_model: vidore/ColSmolVLM-Instruct-500M
+language:
+- en
+tags:
+- colsmolvlm
+- vidore-experimental
+- vidore
+pipeline_tag: visual-document-retrieval
+---
+# ColSmolVLM-Instruct-500M: Visual Retriever based on SmolVLM-Instruct-500M with ColBERT strategy
+
+### This is a version trained with batch_size 32 for 3 epochs
+
+ColSmolVLM is a model based on a novel model architecture and training strategy based on Vision Language Models (VLMs) to efficiently index documents from their visual features.
+It is a SmolVLM extension that generates [ColBERT](https://arxiv.org/abs/2004.12832)- style multi-vector representations of text and images. 
+It was introduced in the paper [ColPali: Efficient Document Retrieval with Vision Language Models](https://arxiv.org/abs/2407.01449) and first released in [this repository](https://github.com/ManuelFay/colpali)
+
+<p align="center"><img width=800 src="https://github.com/illuin-tech/colpali/blob/main/assets/colpali_architecture.webp?raw=true"/></p>
+
+## Version specificity
+
+This version is trained with the commit b983e40 of the Colpali repository. (main branch from the repo)
+
+Data is the same as the ColPali data described in the paper.
+
+
+## Model Training
+
+### Dataset
+Our training dataset of 127,460 query-page pairs is comprised of train sets of openly available academic datasets (63%) and a synthetic dataset made up of pages from web-crawled PDF documents and augmented with VLM-generated (Claude-3 Sonnet) pseudo-questions (37%). 
+Our training set is fully English by design, enabling us to study zero-shot generalization to non-English languages. We explicitly verify no multi-page PDF document is used both [*ViDoRe*](https://huggingface.co/collections/vidore/vidore-benchmark-667173f98e70a1c0fa4db00d) and in the train set to prevent evaluation contamination. 
+A validation set is created with 2% of the samples to tune hyperparameters.
+
+*Note: Multilingual data is present in the pretraining corpus of the language model and most probably in the multimodal training.*
+
+### Parameters
+
+Unless specified otherwise, we train models in `bfloat16` format, use low-rank adapters ([LoRA](https://arxiv.org/abs/2106.09685)) 
+with `alpha=32`  and `r=32` on the transformer layers from the language model, 
+as well as the final randomly initialized projection layer, and use a `paged_adamw_8bit` optimizer. 
+We train on a 4 GPU setup with data parallelism, a learning rate of 5e-4 with linear decay with 2.5% warmup steps, and a batch size of 8.
+
+## Usage
+
+Make sure `colpali-engine` is installed from source or with a version superior to 0.3.5 (main branch from the repo currently).
+`transformers` version must be > 4.46.2.
+
+```bash
+pip install git+https://github.com/illuin-tech/colpali
+```
+
+```python
+import torch
+from PIL import Image
+
+from colpali_engine.models import ColIdefics3, ColIdefics3Processor
+
+model = ColIdefics3.from_pretrained(
+        "vidore/colSmol-500M",
+        torch_dtype=torch.bfloat16,
+        device_map="cuda:0",
+        attn_implementation="flash_attention_2" # or eager
+    ).eval()
+processor = ColIdefics3Processor.from_pretrained("vidore/colSmol-500M")
+
+# Your inputs
+images = [
+    Image.new("RGB", (32, 32), color="white"),
+    Image.new("RGB", (16, 16), color="black"),
+]
+queries = [
+    "Is attention really all you need?",
+    "What is the amount of bananas farmed in Salvador?",
+]
+
+# Process the inputs
+batch_images = processor.process_images(images).to(model.device)
+batch_queries = processor.process_queries(queries).to(model.device)
+
+# Forward pass
+with torch.no_grad():
+    image_embeddings = model(**batch_images)
+    query_embeddings = model(**batch_queries)
+
+scores = processor.score_multi_vector(query_embeddings, image_embeddings)
+```
+
+
+## Limitations
+
+ - **Focus**: The model primarily focuses on PDF-type documents and high-ressources languages, potentially limiting its generalization to other document types or less represented languages.
+ - **Support**: The model relies on multi-vector retreiving derived from the ColBERT late interaction mechanism, which may require engineering efforts to adapt to widely used vector retrieval frameworks that lack native multi-vector support.
+
+## License
+
+ColQwen2's vision language backbone model (Qwen2-VL) is under `apache2.0` license. The adapters attached to the model are under MIT license.
+
+## Contact
+
+- Manuel Faysse: [email protected]
+- Hugues Sibille: [email protected]
+- Tony Wu: [email protected]
+
+## Citation
+
+If you use any datasets or models from this organization in your research, please cite the original dataset as follows:
+
+```bibtex
+@misc{faysse2024colpaliefficientdocumentretrieval,
+  title={ColPali: Efficient Document Retrieval with Vision Language Models}, 
+  author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo},
+  year={2024},
+  eprint={2407.01449},
+  archivePrefix={arXiv},
+  primaryClass={cs.IR},
+  url={https://arxiv.org/abs/2407.01449}, 
+}
+```

adapter_config.json

@@ -0,0 +1,29 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "vidore/ColSmolVLM-Instruct-500M-base",
+  "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": "gaussian",
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "(.*(model.text_model).*(down_proj|gate_proj|up_proj|k_proj|q_proj|v_proj|o_proj).*$|.*(custom_text_proj).*$)",
+  "task_type": "FEATURE_EXTRACTION",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f9d12efede94dc2793a2524694d66f638cd0f818f7b15aef769de0336a8e9cc
+size 69532280

added_tokens.json

@@ -0,0 +1,130 @@
+{
+  "<end_of_utterance>": 49279,
+  "<fake_token_around_image>": 49189,
+  "<global-img>": 49152,
+  "<image>": 49190,
+  "<row_1_col_1>": 49153,
+  "<row_1_col_2>": 49154,
+  "<row_1_col_3>": 49155,
+  "<row_1_col_4>": 49156,
+  "<row_1_col_5>": 49157,
+  "<row_1_col_6>": 49158,
+  "<row_2_col_1>": 49159,
+  "<row_2_col_2>": 49160,
+  "<row_2_col_3>": 49161,
+  "<row_2_col_4>": 49162,
+  "<row_2_col_5>": 49163,
+  "<row_2_col_6>": 49164,
+  "<row_3_col_1>": 49165,
+  "<row_3_col_2>": 49166,
+  "<row_3_col_3>": 49167,
+  "<row_3_col_4>": 49168,
+  "<row_3_col_5>": 49169,
+  "<row_3_col_6>": 49170,
+  "<row_4_col_1>": 49171,
+  "<row_4_col_2>": 49172,
+  "<row_4_col_3>": 49173,
+  "<row_4_col_4>": 49174,
+  "<row_4_col_5>": 49175,
+  "<row_4_col_6>": 49176,
+  "<row_5_col_1>": 49177,
+  "<row_5_col_2>": 49178,
+  "<row_5_col_3>": 49179,
+  "<row_5_col_4>": 49180,
+  "<row_5_col_5>": 49181,
+  "<row_5_col_6>": 49182,
+  "<row_6_col_1>": 49183,
+  "<row_6_col_2>": 49184,
+  "<row_6_col_3>": 49185,
+  "<row_6_col_4>": 49186,
+  "<row_6_col_5>": 49187,
+  "<row_6_col_6>": 49188,
+  "<|reserved_special_token_0|>": 49191,
+  "<|reserved_special_token_10|>": 49201,
+  "<|reserved_special_token_11|>": 49202,
+  "<|reserved_special_token_12|>": 49203,
+  "<|reserved_special_token_13|>": 49204,
+  "<|reserved_special_token_14|>": 49205,
+  "<|reserved_special_token_15|>": 49206,
+  "<|reserved_special_token_16|>": 49207,
+  "<|reserved_special_token_17|>": 49208,
+  "<|reserved_special_token_18|>": 49209,
+  "<|reserved_special_token_19|>": 49210,
+  "<|reserved_special_token_1|>": 49192,
+  "<|reserved_special_token_20|>": 49211,
+  "<|reserved_special_token_21|>": 49212,
+  "<|reserved_special_token_22|>": 49213,
+  "<|reserved_special_token_23|>": 49214,
+  "<|reserved_special_token_24|>": 49215,
+  "<|reserved_special_token_25|>": 49216,
+  "<|reserved_special_token_26|>": 49217,
+  "<|reserved_special_token_27|>": 49218,
+  "<|reserved_special_token_28|>": 49219,
+  "<|reserved_special_token_29|>": 49220,
+  "<|reserved_special_token_2|>": 49193,
+  "<|reserved_special_token_30|>": 49221,
+  "<|reserved_special_token_31|>": 49222,
+  "<|reserved_special_token_32|>": 49223,
+  "<|reserved_special_token_33|>": 49224,
+  "<|reserved_special_token_34|>": 49225,
+  "<|reserved_special_token_35|>": 49226,
+  "<|reserved_special_token_36|>": 49227,
+  "<|reserved_special_token_37|>": 49228,
+  "<|reserved_special_token_38|>": 49229,
+  "<|reserved_special_token_39|>": 49230,
+  "<|reserved_special_token_3|>": 49194,
+  "<|reserved_special_token_40|>": 49231,
+  "<|reserved_special_token_41|>": 49232,
+  "<|reserved_special_token_42|>": 49233,
+  "<|reserved_special_token_43|>": 49234,
+  "<|reserved_special_token_44|>": 49235,
+  "<|reserved_special_token_45|>": 49236,
+  "<|reserved_special_token_46|>": 49237,
+  "<|reserved_special_token_47|>": 49238,
+  "<|reserved_special_token_48|>": 49239,
+  "<|reserved_special_token_49|>": 49240,
+  "<|reserved_special_token_4|>": 49195,
+  "<|reserved_special_token_50|>": 49241,
+  "<|reserved_special_token_51|>": 49242,
+  "<|reserved_special_token_52|>": 49243,
+  "<|reserved_special_token_53|>": 49244,
+  "<|reserved_special_token_54|>": 49245,
+  "<|reserved_special_token_55|>": 49246,
+  "<|reserved_special_token_56|>": 49247,
+  "<|reserved_special_token_57|>": 49248,
+  "<|reserved_special_token_58|>": 49249,
+  "<|reserved_special_token_59|>": 49250,
+  "<|reserved_special_token_5|>": 49196,
+  "<|reserved_special_token_60|>": 49251,
+  "<|reserved_special_token_61|>": 49252,
+  "<|reserved_special_token_62|>": 49253,
+  "<|reserved_special_token_63|>": 49254,
+  "<|reserved_special_token_64|>": 49255,
+  "<|reserved_special_token_65|>": 49256,
+  "<|reserved_special_token_66|>": 49257,
+  "<|reserved_special_token_67|>": 49258,
+  "<|reserved_special_token_68|>": 49259,
+  "<|reserved_special_token_69|>": 49260,
+  "<|reserved_special_token_6|>": 49197,
+  "<|reserved_special_token_70|>": 49261,
+  "<|reserved_special_token_71|>": 49262,
+  "<|reserved_special_token_72|>": 49263,
+  "<|reserved_special_token_73|>": 49264,
+  "<|reserved_special_token_74|>": 49265,
+  "<|reserved_special_token_75|>": 49266,
+  "<|reserved_special_token_76|>": 49267,
+  "<|reserved_special_token_77|>": 49268,
+  "<|reserved_special_token_78|>": 49269,
+  "<|reserved_special_token_79|>": 49270,
+  "<|reserved_special_token_7|>": 49198,
+  "<|reserved_special_token_80|>": 49271,
+  "<|reserved_special_token_81|>": 49272,
+  "<|reserved_special_token_82|>": 49273,
+  "<|reserved_special_token_83|>": 49274,
+  "<|reserved_special_token_84|>": 49275,
+  "<|reserved_special_token_85|>": 49276,
+  "<|reserved_special_token_86|>": 49277,
+  "<|reserved_special_token_87|>": 49278,
+  "<|reserved_special_token_8|>": 49199,
+  "<|reserved_special_token_9|>": 49200
+}

chat_template.json

@@ -0,0 +1,3 @@
+{
+  "chat_template": "<|im_start|>{% for message in messages %}{{message['role'] | capitalize}}{% if message['content'][0]['type'] == 'image' %}{{':'}}{% else %}{{': '}}{% endif %}{% for line in message['content'] %}{% if line['type'] == 'text' %}{{line['text']}}{% elif line['type'] == 'image' %}{{ '<image>' }}{% endif %}{% endfor %}<end_of_utterance>\n{% endfor %}{% if add_generation_prompt %}{{ 'Assistant:' }}{% endif %}"
+}

config.json

@@ -0,0 +1,134 @@
+{
+  "_name_or_path": "models/SmolVLM-Instruct-500M",
+  "architectures": [
+    "ColIdefics3"
+  ],
+  "image_token_id": 49190,
+  "model_type": "idefics3",
+  "pad_token_id": 128002,
+  "scale_factor": 4,
+  "text_config": {
+    "_flash_attn_2_enabled": true,
+    "_name_or_path": "None",
+    "architectures": [
+      "VLlama3ForCausalLM"
+    ],
+    "head_dim": 64,
+    "hidden_size": 960,
+    "intermediate_size": 2560,
+    "is_llama_config": true,
+    "max_position_embeddings": 8192,
+    "model_type": "llama",
+    "neftune_noise_alpha": 0.0,
+    "num_attention_heads": 15,
+    "num_key_value_heads": 5,
+    "pad_token_id": 2,
+    "perceiver_config": {
+      "_attn_implementation_autoset": false,
+      "_name_or_path": "",
+      "add_cross_attention": false,
+      "architectures": null,
+      "attention_dropout": 0.0,
+      "bad_words_ids": null,
+      "begin_suppress_tokens": null,
+      "bos_token_id": null,
+      "chunk_size_feed_forward": 0,
+      "cross_attention_hidden_size": null,
+      "decoder_start_token_id": null,
+      "diversity_penalty": 0.0,
+      "do_sample": false,
+      "early_stopping": false,
+      "encoder_no_repeat_ngram_size": 0,
+      "eos_token_id": null,
+      "exponential_decay_length_penalty": null,
+      "finetuning_task": null,
+      "forced_bos_token_id": null,
+      "forced_eos_token_id": null,
+      "hidden_act": "silu",
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "is_decoder": false,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "length_penalty": 1.0,
+      "max_length": 20,
+      "min_length": 0,
+      "model_type": "vllama3",
+      "no_repeat_ngram_size": 0,
+      "num_beam_groups": 1,
+      "num_beams": 1,
+      "num_key_value_heads": 1,
+      "num_return_sequences": 1,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "output_scores": false,
+      "pad_token_id": null,
+      "prefix": null,
+      "problem_type": null,
+      "pruned_heads": {},
+      "qk_layer_norms_perceiver": false,
+      "remove_invalid_values": false,
+      "repetition_penalty": 1.0,
+      "resampler_depth": 6,
+      "resampler_head_dim": 96,
+      "resampler_n_heads": 16,
+      "resampler_n_latents": 64,
+      "return_dict": true,
+      "return_dict_in_generate": false,
+      "sep_token_id": null,
+      "suppress_tokens": null,
+      "task_specific_params": null,
+      "temperature": 1.0,
+      "tf_legacy_loss": false,
+      "tie_encoder_decoder": false,
+      "tie_word_embeddings": true,
+      "tokenizer_class": null,
+      "top_k": 50,
+      "top_p": 1.0,
+      "torch_dtype": null,
+      "torchscript": false,
+      "transformers_version": "4.46.0",
+      "typical_p": 1.0,
+      "use_bfloat16": false
+    },
+    "pixel_shuffle_factor": 4,
+    "qk_layer_norms": false,
+    "rms_norm_eps": 1e-05,
+    "rope_interleaved": false,
+    "rope_theta": 100000,
+    "torch_dtype": "bfloat16",
+    "transformers.js_config": {
+      "kv_cache_dtype": {
+        "fp16": "float16",
+        "q4f16": "float16"
+      }
+    },
+    "use_resampler": false,
+    "vocab_size": 49280
+  },
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.48.1",
+  "use_cache": true,
+  "vision_config": {
+    "hidden_size": 768,
+    "image_size": 512,
+    "max_image_size": {
+      "longest_edge": 512
+    },
+    "model_type": "idefics3_vision",
+    "num_attention_heads": 12,
+    "patch_size": 16,
+    "size": {
+      "longest_edge": 2048
+    },
+    "tie_word_embeddings": false,
+    "use_base_siglip": false
+  },
+  "vocab_size": 49280
+}

git_hash.txt

@@ -0,0 +1 @@
+8b4f75e476bddc6c34a02722761bd3ada6ac0d3d

preprocessor_config.json

@@ -0,0 +1,28 @@
+{
+  "do_convert_rgb": true,
+  "do_image_splitting": true,
+  "do_normalize": true,
+  "do_pad": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "Idefics3ImageProcessor",
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "max_image_size": {
+    "longest_edge": 512
+  },
+  "processor_class": "ColIdefics3Processor",
+  "resample": 1,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "longest_edge": 2048
+  }
+}

processor_config.json

@@ -0,0 +1,4 @@
+{
+  "image_seq_len": 64,
+  "processor_class": "Idefics3Processor"
+}

results.json

@@ -0,0 +1,465 @@
+{"vidore/restaurant_esg_reports_beir": {"ndcg_at_1": 0.44872, "ndcg_at_3": 0.51996, "ndcg_at_5": 0.52232, "ndcg_at_10": 0.57763, "ndcg_at_20": 0.62215, "ndcg_at_50": 0.64665, "ndcg_at_100": 0.65281, "map_at_1": 0.30481, "map_at_3": 0.44263, "map_at_5": 0.46343, "map_at_10": 0.49741, "map_at_20": 0.51592, "map_at_50": 0.52237, "map_at_100": 0.52398, "recall_at_1": 0.30481, "recall_at_3": 0.54615, "recall_at_5": 0.57807, "recall_at_10": 0.71877, "recall_at_20": 0.86108, "recall_at_50": 0.95162, "recall_at_100": 0.97216, "precision_at_1": 0.46154, "precision_at_3": 0.30769, "precision_at_5": 0.21538, "precision_at_10": 0.14038, "precision_at_20": 0.08654, "precision_at_50": 0.04154, "precision_at_100": 0.0225, "mrr_at_1": 0.4423076923076923, "mrr_at_3": 0.5448717948717949, "mrr_at_5": 0.5583333333333333, "mrr_at_10": 0.5792353479853479, "mrr_at_20": 0.5826894961763383, "mrr_at_50": 0.5850171075017945, "mrr_at_100": 0.5850171075017945, "naucs_at_1_max": 0.24419022886817585, "naucs_at_1_std": 0.20913530757518103, "naucs_at_1_diff1": 0.18956477760151022, "naucs_at_3_max": 0.021538678019016246, "naucs_at_3_std": 0.10725557234497837, "naucs_at_3_diff1": -0.04875457481754448, "naucs_at_5_max": 0.014645022986549194, "naucs_at_5_std": 0.044129824597325146, "naucs_at_5_diff1": -0.11174495404454246, "naucs_at_10_max": -0.11448340055341263, "naucs_at_10_std": -0.10902704168644266, "naucs_at_10_diff1": -0.30406392580200137, "naucs_at_20_max": -0.2404702117925663, "naucs_at_20_std": -0.22313364654433895, "naucs_at_20_diff1": -0.31695759183309535, "naucs_at_50_max": -0.31937538050402253, "naucs_at_50_std": -0.2759760110803978, "naucs_at_50_diff1": -0.24888406251943948, "naucs_at_100_max": -0.259723966830786, "naucs_at_100_std": -0.21333793632671658, "naucs_at_100_diff1": -0.23851384327512057},
+        "vidore/synthetics_economics_macro_economy_2024_filtered_v1.0_multilingual": {
+            "ndcg_at_1": 0.39224,
+            "ndcg_at_3": 0.37476,
+            "ndcg_at_5": 0.36148,
+            "ndcg_at_10": 0.36794,
+            "ndcg_at_20": 0.40776,
+            "ndcg_at_50": 0.48463,
+            "ndcg_at_100": 0.53033,
+            "map_at_1": 0.04401,
+            "map_at_3": 0.0993,
+            "map_at_5": 0.13005,
+            "map_at_10": 0.17704,
+            "map_at_20": 0.21969,
+            "map_at_50": 0.26748,
+            "map_at_100": 0.29042,
+            "recall_at_1": 0.04401,
+            "recall_at_3": 0.13187,
+            "recall_at_5": 0.192,
+            "recall_at_10": 0.3039,
+            "recall_at_20": 0.45606,
+            "recall_at_50": 0.67169,
+            "recall_at_100": 0.80877,
+            "precision_at_1": 0.39224,
+            "precision_at_3": 0.35632,
+            "precision_at_5": 0.325,
+            "precision_at_10": 0.28319,
+            "precision_at_20": 0.2306,
+            "precision_at_50": 0.15836,
+            "precision_at_100": 0.10789,
+            "mrr_at_1": 0.3922413793103448,
+            "mrr_at_3": 0.49856321839080464,
+            "mrr_at_5": 0.5224856321839081,
+            "mrr_at_10": 0.5373648741105638,
+            "mrr_at_20": 0.5419287783917268,
+            "mrr_at_50": 0.5442446092894189,
+            "mrr_at_100": 0.5445144362126614,
+            "naucs_at_1_max": -0.09046956137538088,
+            "naucs_at_1_std": -0.029272928593796467,
+            "naucs_at_1_diff1": -0.01949330501218432,
+            "naucs_at_3_max": 0.019777226299839537,
+            "naucs_at_3_std": 0.05400545367360109,
+            "naucs_at_3_diff1": -0.07318677988609872,
+            "naucs_at_5_max": 0.0035718336913224936,
+            "naucs_at_5_std": 0.027188298117242122,
+            "naucs_at_5_diff1": -0.07827063058686147,
+            "naucs_at_10_max": 0.008049347237839717,
+            "naucs_at_10_std": 0.03145707085203085,
+            "naucs_at_10_diff1": -0.07144093527570813,
+            "naucs_at_20_max": -0.003043019306168713,
+            "naucs_at_20_std": 0.04245224333344333,
+            "naucs_at_20_diff1": -0.0974791685465552,
+            "naucs_at_50_max": -0.008203803722240203,
+            "naucs_at_50_std": 0.004296600594517341,
+            "naucs_at_50_diff1": -0.08202642486940749,
+            "naucs_at_100_max": -0.05980969604715223,
+            "naucs_at_100_std": -0.07215268813689295,
+            "naucs_at_100_diff1": -0.05108060621594025
+        },
+        "vidore/synthetic_axa_filtered_v1.0_multilingual": {
+            "ndcg_at_1": 0.31944,
+            "ndcg_at_3": 0.36735,
+            "ndcg_at_5": 0.37705,
+            "ndcg_at_10": 0.40069,
+            "ndcg_at_20": 0.43513,
+            "ndcg_at_50": 0.49148,
+            "ndcg_at_100": 0.52626,
+            "map_at_1": 0.1883,
+            "map_at_3": 0.24848,
+            "map_at_5": 0.27905,
+            "map_at_10": 0.30821,
+            "map_at_20": 0.32807,
+            "map_at_50": 0.34999,
+            "map_at_100": 0.35742,
+            "recall_at_1": 0.1883,
+            "recall_at_3": 0.30968,
+            "recall_at_5": 0.38218,
+            "recall_at_10": 0.47556,
+            "recall_at_20": 0.56898,
+            "recall_at_50": 0.72054,
+            "recall_at_100": 0.86053,
+            "precision_at_1": 0.31944,
+            "precision_at_3": 0.25,
+            "precision_at_5": 0.20278,
+            "precision_at_10": 0.15,
+            "precision_at_20": 0.10694,
+            "precision_at_50": 0.06556,
+            "precision_at_100": 0.03972,
+            "mrr_at_1": 0.3194444444444444,
+            "mrr_at_3": 0.41435185185185186,
+            "mrr_at_5": 0.44143518518518515,
+            "mrr_at_10": 0.45383597883597887,
+            "mrr_at_20": 0.4603018758656014,
+            "mrr_at_50": 0.46184289024143715,
+            "mrr_at_100": 0.46279641774158087,
+            "naucs_at_1_max": 0.25256807521642977,
+            "naucs_at_1_std": 0.5222755178306032,
+            "naucs_at_1_diff1": 0.22322127062494312,
+            "naucs_at_3_max": 0.01027997935801778,
+            "naucs_at_3_std": 0.26082012822073436,
+            "naucs_at_3_diff1": 0.18412709220174425,
+            "naucs_at_5_max": -0.023540307718607937,
+            "naucs_at_5_std": 0.20590277989329697,
+            "naucs_at_5_diff1": 0.1394140867278813,
+            "naucs_at_10_max": -0.1530545347893013,
+            "naucs_at_10_std": 0.07281002973646711,
+            "naucs_at_10_diff1": 0.1087825983057161,
+            "naucs_at_20_max": -0.24948473963259538,
+            "naucs_at_20_std": -0.052063417282516324,
+            "naucs_at_20_diff1": 0.04499575326148708,
+            "naucs_at_50_max": -0.3456643349075049,
+            "naucs_at_50_std": -0.14962525206807992,
+            "naucs_at_50_diff1": -0.1032344032141534,
+            "naucs_at_100_max": -0.41065527201035196,
+            "naucs_at_100_std": -0.24631195551976134,
+            "naucs_at_100_diff1": -0.1270004114792933
+        },
+        "vidore/synthetic_mit_biomedical_tissue_interactions_unfiltered": {
+            "ndcg_at_1": 0.475,
+            "ndcg_at_3": 0.50399,
+            "ndcg_at_5": 0.5426,
+            "ndcg_at_10": 0.56572,
+            "ndcg_at_20": 0.59334,
+            "ndcg_at_50": 0.62054,
+            "ndcg_at_100": 0.63584,
+            "map_at_1": 0.29844,
+            "map_at_3": 0.40685,
+            "map_at_5": 0.44582,
+            "map_at_10": 0.46931,
+            "map_at_20": 0.48245,
+            "map_at_50": 0.49251,
+            "map_at_100": 0.49533,
+            "recall_at_1": 0.29844,
+            "recall_at_3": 0.48891,
+            "recall_at_5": 0.59575,
+            "recall_at_10": 0.67602,
+            "recall_at_20": 0.75242,
+            "recall_at_50": 0.83386,
+            "recall_at_100": 0.89769,
+            "precision_at_1": 0.475,
+            "precision_at_3": 0.30833,
+            "precision_at_5": 0.24875,
+            "precision_at_10": 0.15125,
+            "precision_at_20": 0.09344,
+            "precision_at_50": 0.0465,
+            "precision_at_100": 0.026,
+            "mrr_at_1": 0.49375,
+            "mrr_at_3": 0.5958333333333333,
+            "mrr_at_5": 0.6202083333333334,
+            "mrr_at_10": 0.6287375992063492,
+            "mrr_at_20": 0.6311754670660437,
+            "mrr_at_50": 0.6321625072817313,
+            "mrr_at_100": 0.6324052960661868,
+            "naucs_at_1_max": 0.288181487963544,
+            "naucs_at_1_std": -0.05426311792094572,
+            "naucs_at_1_diff1": 0.4584585410956644,
+            "naucs_at_3_max": 0.17191913339472156,
+            "naucs_at_3_std": -0.020066630109774056,
+            "naucs_at_3_diff1": 0.09632913618821975,
+            "naucs_at_5_max": 0.09215639006811155,
+            "naucs_at_5_std": 0.0012666140953723163,
+            "naucs_at_5_diff1": -0.06192925755984746,
+            "naucs_at_10_max": 0.035222797177968716,
+            "naucs_at_10_std": 0.007929567555839194,
+            "naucs_at_10_diff1": -0.15828889663289214,
+            "naucs_at_20_max": -0.013133460063983105,
+            "naucs_at_20_std": 0.045498580990890984,
+            "naucs_at_20_diff1": -0.18372934627970697,
+            "naucs_at_50_max": -0.05231228123361153,
+            "naucs_at_50_std": 0.08122908162109731,
+            "naucs_at_50_diff1": -0.18562366170287978,
+            "naucs_at_100_max": -0.09050436131033135,
+            "naucs_at_100_std": 0.07301259934095748,
+            "naucs_at_100_diff1": -0.18245241325838332
+        },
+        "vidore/synthetic_rse_restaurant_filtered_v1.0": {
+            "ndcg_at_1": 0.26316,
+            "ndcg_at_3": 0.36311,
+            "ndcg_at_5": 0.39176,
+            "ndcg_at_10": 0.4454,
+            "ndcg_at_20": 0.50519,
+            "ndcg_at_50": 0.55061,
+            "ndcg_at_100": 0.56466,
+            "map_at_1": 0.15482,
+            "map_at_3": 0.27015,
+            "map_at_5": 0.29589,
+            "map_at_10": 0.33078,
+            "map_at_20": 0.36126,
+            "map_at_50": 0.3824,
+            "map_at_100": 0.38909,
+            "recall_at_1": 0.15482,
+            "recall_at_3": 0.37362,
+            "recall_at_5": 0.44749,
+            "recall_at_10": 0.60324,
+            "recall_at_20": 0.78193,
+            "recall_at_50": 0.91531,
+            "recall_at_100": 0.95395,
+            "precision_at_1": 0.26316,
+            "precision_at_3": 0.25146,
+            "precision_at_5": 0.20702,
+            "precision_at_10": 0.15088,
+            "precision_at_20": 0.11228,
+            "precision_at_50": 0.06667,
+            "precision_at_100": 0.03789,
+            "mrr_at_1": 0.3157894736842105,
+            "mrr_at_3": 0.4473684210526316,
+            "mrr_at_5": 0.46754385964912276,
+            "mrr_at_10": 0.48647312726260095,
+            "mrr_at_20": 0.49202021175705385,
+            "mrr_at_50": 0.49322536427799585,
+            "mrr_at_100": 0.49322536427799585,
+            "naucs_at_1_max": -0.05405282049792571,
+            "naucs_at_1_std": 0.023445855968560407,
+            "naucs_at_1_diff1": 0.21928586224649782,
+            "naucs_at_3_max": -0.011620395920182198,
+            "naucs_at_3_std": 0.08615056848034938,
+            "naucs_at_3_diff1": -0.15099634717817523,
+            "naucs_at_5_max": -0.1652385842485644,
+            "naucs_at_5_std": -0.1076298647113443,
+            "naucs_at_5_diff1": -0.26773917349447285,
+            "naucs_at_10_max": -0.20948212317741371,
+            "naucs_at_10_std": -0.04217452519607505,
+            "naucs_at_10_diff1": -0.17241123984858953,
+            "naucs_at_20_max": -0.3585661146008277,
+            "naucs_at_20_std": -0.11294869002059522,
+            "naucs_at_20_diff1": -0.27144993416761004,
+            "naucs_at_50_max": -0.33543648981957763,
+            "naucs_at_50_std": -0.014416596192293988,
+            "naucs_at_50_diff1": -0.2783603795291369,
+            "naucs_at_100_max": -0.3125483341345789,
+            "naucs_at_100_std": 0.04759656217556336,
+            "naucs_at_100_diff1": -0.27748680314970026
+        },
+        "vidore/synthetic_rse_restaurant_filtered_v1.0_multilingual": {
+            "ndcg_at_1": 0.3114,
+            "ndcg_at_3": 0.35716,
+            "ndcg_at_5": 0.39093,
+            "ndcg_at_10": 0.45776,
+            "ndcg_at_20": 0.51442,
+            "ndcg_at_50": 0.55435,
+            "ndcg_at_100": 0.57057,
+            "map_at_1": 0.1635,
+            "map_at_3": 0.26015,
+            "map_at_5": 0.29428,
+            "map_at_10": 0.33703,
+            "map_at_20": 0.36703,
+            "map_at_50": 0.38532,
+            "map_at_100": 0.39218,
+            "recall_at_1": 0.1635,
+            "recall_at_3": 0.34212,
+            "recall_at_5": 0.43294,
+            "recall_at_10": 0.62713,
+            "recall_at_20": 0.80024,
+            "recall_at_50": 0.9216,
+            "recall_at_100": 0.97058,
+            "precision_at_1": 0.3114,
+            "precision_at_3": 0.25,
+            "precision_at_5": 0.21228,
+            "precision_at_10": 0.16272,
+            "precision_at_20": 0.11732,
+            "precision_at_50": 0.06509,
+            "precision_at_100": 0.03768,
+            "mrr_at_1": 0.3201754385964912,
+            "mrr_at_3": 0.4276315789473684,
+            "mrr_at_5": 0.45570175438596494,
+            "mrr_at_10": 0.4726991088833194,
+            "mrr_at_20": 0.47990133359766324,
+            "mrr_at_50": 0.4817393870105699,
+            "mrr_at_100": 0.48179098659777314,
+            "naucs_at_1_max": -0.043051520891704526,
+            "naucs_at_1_std": 0.04332255792861466,
+            "naucs_at_1_diff1": 0.25290228804176196,
+            "naucs_at_3_max": -0.015276644061893118,
+            "naucs_at_3_std": 0.05163750744485839,
+            "naucs_at_3_diff1": 0.040189218663360116,
+            "naucs_at_5_max": -0.05235721102136031,
+            "naucs_at_5_std": 0.005811214103192345,
+            "naucs_at_5_diff1": 0.019471817185607386,
+            "naucs_at_10_max": -0.15526718391563657,
+            "naucs_at_10_std": -0.030920730403369213,
+            "naucs_at_10_diff1": -0.009103205213602163,
+            "naucs_at_20_max": -0.16361665459184682,
+            "naucs_at_20_std": -0.006377615107252758,
+            "naucs_at_20_diff1": -0.11234618830619728,
+            "naucs_at_50_max": -0.15363708207480695,
+            "naucs_at_50_std": 0.07280318534401509,
+            "naucs_at_50_diff1": -0.15710162022601212,
+            "naucs_at_100_max": -0.1483886076804903,
+            "naucs_at_100_std": 0.10513102578338192,
+            "naucs_at_100_diff1": -0.15391945667071522
+        },
+        "vidore/synthetic_mit_biomedical_tissue_interactions_unfiltered_multilingual": {
+            "ndcg_at_1": 0.38125,
+            "ndcg_at_3": 0.39476,
+            "ndcg_at_5": 0.4206,
+            "ndcg_at_10": 0.45509,
+            "ndcg_at_20": 0.48714,
+            "ndcg_at_50": 0.51836,
+            "ndcg_at_100": 0.53432,
+            "map_at_1": 0.24209,
+            "map_at_3": 0.31985,
+            "map_at_5": 0.34726,
+            "map_at_10": 0.3711,
+            "map_at_20": 0.38475,
+            "map_at_50": 0.39391,
+            "map_at_100": 0.39676,
+            "recall_at_1": 0.24209,
+            "recall_at_3": 0.38347,
+            "recall_at_5": 0.45975,
+            "recall_at_10": 0.56148,
+            "recall_at_20": 0.65407,
+            "recall_at_50": 0.75821,
+            "recall_at_100": 0.82192,
+            "precision_at_1": 0.38125,
+            "precision_at_3": 0.23333,
+            "precision_at_5": 0.18344,
+            "precision_at_10": 0.12031,
+            "precision_at_20": 0.07758,
+            "precision_at_50": 0.04006,
+            "precision_at_100": 0.02297,
+            "mrr_at_1": 0.3984375,
+            "mrr_at_3": 0.47890625,
+            "mrr_at_5": 0.497734375,
+            "mrr_at_10": 0.5095529513888889,
+            "mrr_at_20": 0.5151586381105779,
+            "mrr_at_50": 0.5174101160297588,
+            "mrr_at_100": 0.5178826588862087,
+            "naucs_at_1_max": 0.13899599795432951,
+            "naucs_at_1_std": 0.0368281555683635,
+            "naucs_at_1_diff1": 0.5053982968982114,
+            "naucs_at_3_max": 0.07054641540082532,
+            "naucs_at_3_std": 0.05000850601457527,
+            "naucs_at_3_diff1": 0.21532701510229157,
+            "naucs_at_5_max": 0.01417845342936614,
+            "naucs_at_5_std": 0.020526726379349457,
+            "naucs_at_5_diff1": 0.10818783419375576,
+            "naucs_at_10_max": -0.020678341996715813,
+            "naucs_at_10_std": 0.02320094009445125,
+            "naucs_at_10_diff1": 0.006460602129137368,
+            "naucs_at_20_max": -0.056359581334902174,
+            "naucs_at_20_std": 0.022492779057438163,
+            "naucs_at_20_diff1": -0.04495830300371242,
+            "naucs_at_50_max": -0.08195205931936514,
+            "naucs_at_50_std": 0.036322084125175555,
+            "naucs_at_50_diff1": -0.07768679112951372,
+            "naucs_at_100_max": -0.10382076960662541,
+            "naucs_at_100_std": 0.017015419651631922,
+            "naucs_at_100_diff1": -0.08434098316520129
+        },
+        "vidore/synthetic_axa_filtered_v1.0": {
+            "ndcg_at_1": 0.5,
+            "ndcg_at_3": 0.56184,
+            "ndcg_at_5": 0.587,
+            "ndcg_at_10": 0.60148,
+            "ndcg_at_20": 0.62635,
+            "ndcg_at_50": 0.67819,
+            "ndcg_at_100": 0.69787,
+            "map_at_1": 0.28611,
+            "map_at_3": 0.36587,
+            "map_at_5": 0.43319,
+            "map_at_10": 0.47842,
+            "map_at_20": 0.50424,
+            "map_at_50": 0.53292,
+            "map_at_100": 0.53674,
+            "recall_at_1": 0.28611,
+            "recall_at_3": 0.42945,
+            "recall_at_5": 0.58465,
+            "recall_at_10": 0.68747,
+            "recall_at_20": 0.75855,
+            "recall_at_50": 0.88111,
+            "recall_at_100": 0.97026,
+            "precision_at_1": 0.5,
+            "precision_at_3": 0.40741,
+            "precision_at_5": 0.33333,
+            "precision_at_10": 0.23333,
+            "precision_at_20": 0.15278,
+            "precision_at_50": 0.08444,
+            "precision_at_100": 0.045,
+            "mrr_at_1": 0.5,
+            "mrr_at_3": 0.6203703703703703,
+            "mrr_at_5": 0.6481481481481481,
+            "mrr_at_10": 0.6574074074074074,
+            "mrr_at_20": 0.6574074074074074,
+            "mrr_at_50": 0.65854119425548,
+            "mrr_at_100": 0.65854119425548,
+            "naucs_at_1_max": 0.42651593299790197,
+            "naucs_at_1_std": 0.5071500249117564,
+            "naucs_at_1_diff1": 0.30239223033901025,
+            "naucs_at_3_max": -0.2757769266874663,
+            "naucs_at_3_std": -0.205926822886879,
+            "naucs_at_3_diff1": 0.06908403486928417,
+            "naucs_at_5_max": -0.3693499917580161,
+            "naucs_at_5_std": -0.2923131576624302,
+            "naucs_at_5_diff1": -0.050910460634029486,
+            "naucs_at_10_max": -0.572254001138292,
+            "naucs_at_10_std": -0.46716738275931796,
+            "naucs_at_10_diff1": -0.0735594830753536,
+            "naucs_at_20_max": -0.6311605275619012,
+            "naucs_at_20_std": -0.493877197928487,
+            "naucs_at_20_diff1": -0.08810700944293344,
+            "naucs_at_50_max": -0.5994854185823998,
+            "naucs_at_50_std": -0.3880377145984835,
+            "naucs_at_50_diff1": -0.2337980752344902,
+            "naucs_at_100_max": -0.5975814841373683,
+            "naucs_at_100_std": -0.3931594317744619,
+            "naucs_at_100_diff1": -0.20608009190940405
+        },
+        "vidore/synthetic_economics_macro_economy_2024_filtered_v1.0": {
+            "ndcg_at_1": 0.56897,
+            "ndcg_at_3": 0.5211,
+            "ndcg_at_5": 0.50298,
+            "ndcg_at_10": 0.52258,
+            "ndcg_at_20": 0.55507,
+            "ndcg_at_50": 0.62454,
+            "ndcg_at_100": 0.65668,
+            "map_at_1": 0.06658,
+            "map_at_3": 0.14611,
+            "map_at_5": 0.19305,
+            "map_at_10": 0.27576,
+            "map_at_20": 0.33378,
+            "map_at_50": 0.39269,
+            "map_at_100": 0.41666,
+            "recall_at_1": 0.06658,
+            "recall_at_3": 0.18538,
+            "recall_at_5": 0.26874,
+            "recall_at_10": 0.44378,
+            "recall_at_20": 0.5979,
+            "recall_at_50": 0.79293,
+            "recall_at_100": 0.89795,
+            "precision_at_1": 0.56897,
+            "precision_at_3": 0.48851,
+            "precision_at_5": 0.44483,
+            "precision_at_10": 0.3931,
+            "precision_at_20": 0.30431,
+            "precision_at_50": 0.19517,
+            "precision_at_100": 0.12379,
+            "mrr_at_1": 0.5862068965517241,
+            "mrr_at_3": 0.675287356321839,
+            "mrr_at_5": 0.6985632183908046,
+            "mrr_at_10": 0.7113711001642037,
+            "mrr_at_20": 0.71252052545156,
+            "mrr_at_50": 0.7133415435139574,
+            "mrr_at_100": 0.7133415435139574,
+            "naucs_at_1_max": -0.40040744598442884,
+            "naucs_at_1_std": -0.2985416538266988,
+            "naucs_at_1_diff1": 0.12036209692812928,
+            "naucs_at_3_max": -0.3377813006714379,
+            "naucs_at_3_std": -0.2501659993033734,
+            "naucs_at_3_diff1": -0.03738351632070548,
+            "naucs_at_5_max": -0.1125861127198914,
+            "naucs_at_5_std": -0.06443718988916795,
+            "naucs_at_5_diff1": -0.03965316546372891,
+            "naucs_at_10_max": -0.009590286755270482,
+            "naucs_at_10_std": 0.006695305574988969,
+            "naucs_at_10_diff1": -0.09088428457053505,
+            "naucs_at_20_max": 0.05383858850417515,
+            "naucs_at_20_std": 0.05335828984185507,
+            "naucs_at_20_diff1": -0.12843644755345873,
+            "naucs_at_50_max": 0.016749758399340033,
+            "naucs_at_50_std": 0.006620482393505167,
+            "naucs_at_50_diff1": -0.09393030796852221,
+            "naucs_at_100_max": -0.069387597095011,
+            "naucs_at_100_std": -0.09518999664037518,
+            "naucs_at_100_diff1": -0.05750748161503184
+        }, "./data_dir/eval_vidore/arxivqa_test_subsampled": {"ndcg_at_1": 0.67, "ndcg_at_3": 0.73664, "ndcg_at_5": 0.74886, "ndcg_at_10": 0.76608, "ndcg_at_20": 0.77508, "ndcg_at_50": 0.78465, "ndcg_at_100": 0.78928, "map_at_1": 0.67, "map_at_3": 0.71967, "map_at_5": 0.72637, "map_at_10": 0.73332, "map_at_20": 0.73575, "map_at_50": 0.7373, "map_at_100": 0.73774, "recall_at_1": 0.67, "recall_at_3": 0.786, "recall_at_5": 0.816, "recall_at_10": 0.87, "recall_at_20": 0.906, "recall_at_50": 0.954, "recall_at_100": 0.982, "precision_at_1": 0.67, "precision_at_3": 0.262, "precision_at_5": 0.1632, "precision_at_10": 0.087, "precision_at_20": 0.0453, "precision_at_50": 0.01908, "precision_at_100": 0.00982, "mrr_at_1": 0.674, "mrr_at_3": 0.723, "mrr_at_5": 0.7291, "mrr_at_10": 0.7366095238095238, "mrr_at_20": 0.7391611043325439, "mrr_at_50": 0.7409444251953623, "mrr_at_100": 0.7413271325461593, "naucs_at_1_max": 0.5444803710788707, "naucs_at_1_std": -0.09109715427197845, "naucs_at_1_diff1": 0.9253203771732317, "naucs_at_3_max": 0.5919376009932287, "naucs_at_3_std": -0.04496538390458086, "naucs_at_3_diff1": 0.8649522103597137, "naucs_at_5_max": 0.6306070025471655, "naucs_at_5_std": 0.001678323187842219, "naucs_at_5_diff1": 0.8645469067046583, "naucs_at_10_max": 0.6801229356439317, "naucs_at_10_std": 0.03928756572613559, "naucs_at_10_diff1": 0.8395171443382958, "naucs_at_20_max": 0.7005979696843283, "naucs_at_20_std": 0.04557283906470355, "naucs_at_20_diff1": 0.8023322804299033, "naucs_at_50_max": 0.7126212803962136, "naucs_at_50_std": -0.09452766613891975, "naucs_at_50_diff1": 0.8312223440100683, "naucs_at_100_max": 0.85968461458657, "naucs_at_100_std": -0.10281149496835222, "naucs_at_100_diff1": 0.896514161220045}, "./data_dir/eval_vidore/docvqa_test_subsampled": {"ndcg_at_1": 0.50554, "ndcg_at_3": 0.55831, "ndcg_at_5": 0.58055, "ndcg_at_10": 0.60677, "ndcg_at_20": 0.62553, "ndcg_at_50": 0.63388, "ndcg_at_100": 0.64093, "map_at_1": 0.50554, "map_at_3": 0.54582, "map_at_5": 0.55835, "map_at_10": 0.569, "map_at_20": 0.5743, "map_at_50": 0.57564, "map_at_100": 0.57622, "recall_at_1": 0.50554, "recall_at_3": 0.59424, "recall_at_5": 0.64745, "recall_at_10": 0.72949, "recall_at_20": 0.80266, "recall_at_50": 0.84479, "recall_at_100": 0.88914, "precision_at_1": 0.50554, "precision_at_3": 0.19808, "precision_at_5": 0.12949, "precision_at_10": 0.07295, "precision_at_20": 0.04013, "precision_at_50": 0.0169, "precision_at_100": 0.00889, "mrr_at_1": 0.5033259423503326, "mrr_at_3": 0.5476718403547672, "mrr_at_5": 0.5574279379157429, "mrr_at_10": 0.5684528208918452, "mrr_at_20": 0.573404311645784, "mrr_at_50": 0.5749334030411379, "mrr_at_100": 0.575603354450426, "naucs_at_1_max": 0.2625490776539377, "naucs_at_1_std": 0.3310850636640096, "naucs_at_1_diff1": 0.8285865433933254, "naucs_at_3_max": 0.18953190824386826, "naucs_at_3_std": 0.38629715857674524, "naucs_at_3_diff1": 0.769918430991528, "naucs_at_5_max": 0.14915660669648698, "naucs_at_5_std": 0.39060413735358934, "naucs_at_5_diff1": 0.7373299633405666, "naucs_at_10_max": 0.008734548704545675, "naucs_at_10_std": 0.5053066426323437, "naucs_at_10_diff1": 0.7260812769949216, "naucs_at_20_max": -0.1110192872274276, "naucs_at_20_std": 0.6213072109985829, "naucs_at_20_diff1": 0.6964830158995551, "naucs_at_50_max": -0.16049662657697744, "naucs_at_50_std": 0.7364897688356142, "naucs_at_50_diff1": 0.7182771454403309, "naucs_at_100_max": -0.19221684927376187, "naucs_at_100_std": 0.8505984612548488, "naucs_at_100_diff1": 0.7419613287710981}, "./data_dir/eval_vidore/syntheticDocQA_energy_test": {"ndcg_at_1": 0.88, "ndcg_at_3": 0.93417, "ndcg_at_5": 0.93847, "ndcg_at_10": 0.93847, "ndcg_at_20": 0.93847, "ndcg_at_50": 0.94063, "ndcg_at_100": 0.9422, "map_at_1": 0.88, "map_at_3": 0.92167, "map_at_5": 0.92417, "map_at_10": 0.92417, "map_at_20": 0.92417, "map_at_50": 0.92458, "map_at_100": 0.92471, "recall_at_1": 0.88, "recall_at_3": 0.97, "recall_at_5": 0.98, "recall_at_10": 0.98, "recall_at_20": 0.98, "recall_at_50": 0.99, "recall_at_100": 1.0, "precision_at_1": 0.88, "precision_at_3": 0.32333, "precision_at_5": 0.196, "precision_at_10": 0.098, "precision_at_20": 0.049, "precision_at_50": 0.0198, "precision_at_100": 0.01, "mrr_at_1": 0.89, "mrr_at_3": 0.93, "mrr_at_5": 0.9325, "mrr_at_10": 0.9325, "mrr_at_20": 0.9325, "mrr_at_50": 0.9329166666666667, "mrr_at_100": 0.9330448717948718, "naucs_at_1_max": 0.36497772828507796, "naucs_at_1_std": -0.272231944002546, "naucs_at_1_diff1": 0.9777282850779508, "naucs_at_3_max": 0.693432928727046, "naucs_at_3_std": -0.15452847805789072, "naucs_at_3_diff1": 0.9564270152505466, "naucs_at_5_max": 0.6790382819794609, "naucs_at_5_std": -0.5929038281979465, "naucs_at_5_diff1": 0.9346405228758136, "naucs_at_10_max": 0.6790382819794609, "naucs_at_10_std": -0.5929038281979465, "naucs_at_10_diff1": 0.9346405228758136, "naucs_at_20_max": 0.6790382819794609, "naucs_at_20_std": -0.5929038281979465, "naucs_at_20_diff1": 0.9346405228758136, "naucs_at_50_max": 1.0, "naucs_at_50_std": 0.554154995331464, "naucs_at_50_diff1": 0.8692810457516374, "naucs_at_100_max": NaN, "naucs_at_100_std": NaN, "naucs_at_100_diff1": NaN}, "./data_dir/eval_vidore/tatdqa_test": {"ndcg_at_1": 0.64581, "ndcg_at_3": 0.73981, "ndcg_at_5": 0.7668, "ndcg_at_10": 0.787, "ndcg_at_20": 0.79364, "ndcg_at_50": 0.79698, "ndcg_at_100": 0.79864, "map_at_1": 0.64581, "map_at_3": 0.71648, "map_at_5": 0.73158, "map_at_10": 0.74001, "map_at_20": 0.74185, "map_at_50": 0.74242, "map_at_100": 0.74256, "recall_at_1": 0.64581, "recall_at_3": 0.80741, "recall_at_5": 0.87242, "recall_at_10": 0.93439, "recall_at_20": 0.96051, "recall_at_50": 0.97691, "recall_at_100": 0.98724, "precision_at_1": 0.64581, "precision_at_3": 0.26914, "precision_at_5": 0.17448, "precision_at_10": 0.09344, "precision_at_20": 0.04803, "precision_at_50": 0.01954, "precision_at_100": 0.00987, "mrr_at_1": 0.6433778857837181, "mrr_at_3": 0.7150668286755771, "mrr_at_5": 0.7304981773997571, "mrr_at_10": 0.7387049412717699, "mrr_at_20": 0.7404579555191415, "mrr_at_50": 0.7410660643728302, "mrr_at_100": 0.7412038533898355, "naucs_at_1_max": 0.13431737923823708, "naucs_at_1_std": -0.3067075571650227, "naucs_at_1_diff1": 0.8067592167032371, "naucs_at_3_max": 0.18727434840097998, "naucs_at_3_std": -0.27607320609227726, "naucs_at_3_diff1": 0.6950163190033429, "naucs_at_5_max": 0.2109382115959134, "naucs_at_5_std": -0.22542299636024854, "naucs_at_5_diff1": 0.6560772753501689, "naucs_at_10_max": 0.3049581820340452, "naucs_at_10_std": 0.10695854160572203, "naucs_at_10_diff1": 0.5979569201987198, "naucs_at_20_max": 0.3524427577390326, "naucs_at_20_std": 0.2838037847480218, "naucs_at_20_diff1": 0.5748977828288308, "naucs_at_50_max": 0.39877531195470334, "naucs_at_50_std": 0.47760692333540594, "naucs_at_50_diff1": 0.48464495157589027, "naucs_at_100_max": 0.3531041790707774, "naucs_at_100_std": 0.4303621036969795, "naucs_at_100_diff1": 0.42655633089874984}, "./data_dir/eval_vidore/infovqa_test_subsampled": {"ndcg_at_1": 0.80769, "ndcg_at_3": 0.86119, "ndcg_at_5": 0.86799, "ndcg_at_10": 0.87508, "ndcg_at_20": 0.8788, "ndcg_at_50": 0.88321, "ndcg_at_100": 0.88418, "map_at_1": 0.80769, "map_at_3": 0.84818, "map_at_5": 0.85202, "map_at_10": 0.85488, "map_at_20": 0.85598, "map_at_50": 0.85668, "map_at_100": 0.85677, "recall_at_1": 0.80769, "recall_at_3": 0.89879, "recall_at_5": 0.91498, "recall_at_10": 0.93725, "recall_at_20": 0.95142, "recall_at_50": 0.97368, "recall_at_100": 0.97976, "precision_at_1": 0.80769, "precision_at_3": 0.2996, "precision_at_5": 0.183, "precision_at_10": 0.09372, "precision_at_20": 0.04757, "precision_at_50": 0.01947, "precision_at_100": 0.0098, "mrr_at_1": 0.8117408906882592, "mrr_at_3": 0.8488529014844804, "mrr_at_5": 0.8532051282051283, "mrr_at_10": 0.8560359231411864, "mrr_at_20": 0.8570996255206782, "mrr_at_50": 0.8577900744824616, "mrr_at_100": 0.8578711146551276, "naucs_at_1_max": 0.5181292739994149, "naucs_at_1_std": -0.12587128411667012, "naucs_at_1_diff1": 0.9118360487145103, "naucs_at_3_max": 0.5428075702339784, "naucs_at_3_std": -0.133844833983956, "naucs_at_3_diff1": 0.8487810208609975, "naucs_at_5_max": 0.570342039887235, "naucs_at_5_std": -0.1629741497251075, "naucs_at_5_diff1": 0.8403883247563, "naucs_at_10_max": 0.7076634965650749, "naucs_at_10_std": 0.14379204929117678, "naucs_at_10_diff1": 0.8238016143503649, "naucs_at_20_max": 0.8079586920995654, "naucs_at_20_std": 0.21002182843283598, "naucs_at_20_diff1": 0.825877668290874, "naucs_at_50_max": 0.8233669400006994, "naucs_at_50_std": 0.18166589601039382, "naucs_at_50_diff1": 0.8057881231371927, "naucs_at_100_max": 0.8922124708224929, "naucs_at_100_std": 0.6406492513828425, "naucs_at_100_diff1": 0.7867049962977823}, "./data_dir/eval_vidore/syntheticDocQA_healthcare_industry_test": {"ndcg_at_1": 0.95, "ndcg_at_3": 0.97893, "ndcg_at_5": 0.97893, "ndcg_at_10": 0.97893, "ndcg_at_20": 0.97893, "ndcg_at_50": 0.97893, "ndcg_at_100": 0.97893, "map_at_1": 0.95, "map_at_3": 0.97167, "map_at_5": 0.97167, "map_at_10": 0.97167, "map_at_20": 0.97167, "map_at_50": 0.97167, "map_at_100": 0.97167, "recall_at_1": 0.95, "recall_at_3": 1.0, "recall_at_5": 1.0, "recall_at_10": 1.0, "recall_at_20": 1.0, "recall_at_50": 1.0, "recall_at_100": 1.0, "precision_at_1": 0.95, "precision_at_3": 0.33333, "precision_at_5": 0.2, "precision_at_10": 0.1, "precision_at_20": 0.05, "precision_at_50": 0.02, "precision_at_100": 0.01, "mrr_at_1": 0.95, "mrr_at_3": 0.9716666666666667, "mrr_at_5": 0.9716666666666667, "mrr_at_10": 0.9716666666666667, "mrr_at_20": 0.9716666666666667, "mrr_at_50": 0.9716666666666667, "mrr_at_100": 0.9716666666666667, "naucs_at_1_max": 0.5808590102707767, "naucs_at_1_std": -0.1164332399626522, "naucs_at_1_diff1": 0.9477124183006508, "naucs_at_3_max": 1.0, "naucs_at_3_std": 1.0, "naucs_at_3_diff1": 1.0, "naucs_at_5_max": 1.0, "naucs_at_5_std": 1.0, "naucs_at_5_diff1": 1.0, "naucs_at_10_max": 1.0, "naucs_at_10_std": 1.0, "naucs_at_10_diff1": 1.0, "naucs_at_20_max": 1.0, "naucs_at_20_std": 1.0, "naucs_at_20_diff1": 1.0, "naucs_at_50_max": NaN, "naucs_at_50_std": NaN, "naucs_at_50_diff1": NaN, "naucs_at_100_max": NaN, "naucs_at_100_std": NaN, "naucs_at_100_diff1": NaN}, "./data_dir/eval_vidore/tabfquad_test_subsampled": {"ndcg_at_1": 0.68929, "ndcg_at_3": 0.74413, "ndcg_at_5": 0.75167, "ndcg_at_10": 0.77118, "ndcg_at_20": 0.78112, "ndcg_at_50": 0.79873, "ndcg_at_100": 0.80054, "map_at_1": 0.68929, "map_at_3": 0.73095, "map_at_5": 0.73524, "map_at_10": 0.74323, "map_at_20": 0.74596, "map_at_50": 0.74876, "map_at_100": 0.74894, "recall_at_1": 0.68929, "recall_at_3": 0.78214, "recall_at_5": 0.8, "recall_at_10": 0.86071, "recall_at_20": 0.9, "recall_at_50": 0.98929, "recall_at_100": 1.0, "precision_at_1": 0.68929, "precision_at_3": 0.26071, "precision_at_5": 0.16, "precision_at_10": 0.08607, "precision_at_20": 0.045, "precision_at_50": 0.01979, "precision_at_100": 0.01, "mrr_at_1": 0.6857142857142857, "mrr_at_3": 0.7273809523809524, "mrr_at_5": 0.7332738095238095, "mrr_at_10": 0.7413988095238095, "mrr_at_20": 0.7440813863424158, "mrr_at_50": 0.7468722884841633, "mrr_at_100": 0.7470525854039923, "naucs_at_1_max": 0.2870009874897007, "naucs_at_1_std": 0.04204064432130546, "naucs_at_1_diff1": 0.7891552245752849, "naucs_at_3_max": 0.29717890310145373, "naucs_at_3_std": 0.10697806369788107, "naucs_at_3_diff1": 0.6949610753710974, "naucs_at_5_max": 0.2523369487655202, "naucs_at_5_std": 0.06743256743256658, "naucs_at_5_diff1": 0.6558976737548161, "naucs_at_10_max": 0.24856446841830593, "naucs_at_10_std": 0.02682144721469618, "naucs_at_10_diff1": 0.6389296278803848, "naucs_at_20_max": 0.16003067893824288, "naucs_at_20_std": -0.11236161131119002, "naucs_at_20_diff1": 0.601724022942509, "naucs_at_50_max": 0.8202614379084879, "naucs_at_50_std": 0.7424525365701868, "naucs_at_50_diff1": 0.7587924058512231, "naucs_at_100_max": 1.0, "naucs_at_100_std": 1.0, "naucs_at_100_diff1": 1.0}, "./data_dir/eval_vidore/syntheticDocQA_government_reports_test": {"ndcg_at_1": 0.87, "ndcg_at_3": 0.94178, "ndcg_at_5": 0.94609, "ndcg_at_10": 0.94609, "ndcg_at_20": 0.94609, "ndcg_at_50": 0.94609, "ndcg_at_100": 0.94609, "map_at_1": 0.87, "map_at_3": 0.925, "map_at_5": 0.9275, "map_at_10": 0.9275, "map_at_20": 0.9275, "map_at_50": 0.9275, "map_at_100": 0.9275, "recall_at_1": 0.87, "recall_at_3": 0.99, "recall_at_5": 1.0, "recall_at_10": 1.0, "recall_at_20": 1.0, "recall_at_50": 1.0, "recall_at_100": 1.0, "precision_at_1": 0.87, "precision_at_3": 0.33, "precision_at_5": 0.2, "precision_at_10": 0.1, "precision_at_20": 0.05, "precision_at_50": 0.02, "precision_at_100": 0.01, "mrr_at_1": 0.89, "mrr_at_3": 0.935, "mrr_at_5": 0.9375, "mrr_at_10": 0.9375, "mrr_at_20": 0.9375, "mrr_at_50": 0.9375, "mrr_at_100": 0.9375, "naucs_at_1_max": 0.5187736058653628, "naucs_at_1_std": 0.22202473524402008, "naucs_at_1_diff1": 0.7872694956676297, "naucs_at_3_max": 0.5541549953314585, "naucs_at_3_std": 0.35807656395889226, "naucs_at_3_diff1": 1.0, "naucs_at_5_max": 1.0, "naucs_at_5_std": 1.0, "naucs_at_5_diff1": 1.0, "naucs_at_10_max": 1.0, "naucs_at_10_std": 1.0, "naucs_at_10_diff1": 1.0, "naucs_at_20_max": 1.0, "naucs_at_20_std": 1.0, "naucs_at_20_diff1": 1.0, "naucs_at_50_max": NaN, "naucs_at_50_std": NaN, "naucs_at_50_diff1": NaN, "naucs_at_100_max": NaN, "naucs_at_100_std": NaN, "naucs_at_100_diff1": NaN}, "./data_dir/eval_vidore/shiftproject_test": {"ndcg_at_1": 0.48, "ndcg_at_3": 0.62226, "ndcg_at_5": 0.66313, "ndcg_at_10": 0.70227, "ndcg_at_20": 0.70762, "ndcg_at_50": 0.71377, "ndcg_at_100": 0.71377, "map_at_1": 0.48, "map_at_3": 0.58833, "map_at_5": 0.61083, "map_at_10": 0.62718, "map_at_20": 0.6288, "map_at_50": 0.62987, "map_at_100": 0.62987, "recall_at_1": 0.48, "recall_at_3": 0.72, "recall_at_5": 0.82, "recall_at_10": 0.94, "recall_at_20": 0.96, "recall_at_50": 0.99, "recall_at_100": 0.99, "precision_at_1": 0.48, "precision_at_3": 0.24, "precision_at_5": 0.164, "precision_at_10": 0.094, "precision_at_20": 0.048, "precision_at_50": 0.0198, "precision_at_100": 0.0099, "mrr_at_1": 0.51, "mrr_at_3": 0.6133333333333334, "mrr_at_5": 0.6333333333333333, "mrr_at_10": 0.6481230158730159, "mrr_at_20": 0.6503713924963925, "mrr_at_50": 0.6509864903528697, "mrr_at_100": 0.6509864903528697, "naucs_at_1_max": 0.01784958187784493, "naucs_at_1_std": -0.21715888641896902, "naucs_at_1_diff1": 0.7087302847464805, "naucs_at_3_max": 0.08859890109890145, "naucs_at_3_std": -0.28323412698412614, "naucs_at_3_diff1": 0.4830586080586077, "naucs_at_5_max": 0.10656459962550972, "naucs_at_5_std": -0.10389360061680472, "naucs_at_5_diff1": 0.40610199361163074, "naucs_at_10_max": -0.18744164332399568, "naucs_at_10_std": -0.024042950513536908, "naucs_at_10_diff1": 0.6868969810146294, "naucs_at_20_max": -0.22980859010270466, "naucs_at_20_std": -0.1562791783379993, "naucs_at_20_diff1": 0.7112511671335194, "naucs_at_50_max": -0.17133520074697067, "naucs_at_50_std": 0.12278244631185525, "naucs_at_50_diff1": 1.0, "naucs_at_100_max": -0.17133520074697067, "naucs_at_100_std": 0.12278244631185525, "naucs_at_100_diff1": 1.0}, "./data_dir/eval_vidore/syntheticDocQA_artificial_intelligence_test": {"ndcg_at_1": 0.97, "ndcg_at_3": 0.98762, "ndcg_at_5": 0.98762, "ndcg_at_10": 0.98762, "ndcg_at_20": 0.98762, "ndcg_at_50": 0.98762, "ndcg_at_100": 0.98762, "map_at_1": 0.97, "map_at_3": 0.98333, "map_at_5": 0.98333, "map_at_10": 0.98333, "map_at_20": 0.98333, "map_at_50": 0.98333, "map_at_100": 0.98333, "recall_at_1": 0.97, "recall_at_3": 1.0, "recall_at_5": 1.0, "recall_at_10": 1.0, "recall_at_20": 1.0, "recall_at_50": 1.0, "recall_at_100": 1.0, "precision_at_1": 0.97, "precision_at_3": 0.33333, "precision_at_5": 0.2, "precision_at_10": 0.1, "precision_at_20": 0.05, "precision_at_50": 0.02, "precision_at_100": 0.01, "mrr_at_1": 0.97, "mrr_at_3": 0.9833333333333333, "mrr_at_5": 0.9833333333333333, "mrr_at_10": 0.9833333333333333, "mrr_at_20": 0.9833333333333333, "mrr_at_50": 0.9833333333333333, "mrr_at_100": 0.9833333333333333, "naucs_at_1_max": 0.912854030501093, "naucs_at_1_std": 0.6934329287270453, "naucs_at_1_diff1": 0.912854030501093, "naucs_at_3_max": 1.0, "naucs_at_3_std": 1.0, "naucs_at_3_diff1": 1.0, "naucs_at_5_max": 1.0, "naucs_at_5_std": 1.0, "naucs_at_5_diff1": 1.0, "naucs_at_10_max": 1.0, "naucs_at_10_std": 1.0, "naucs_at_10_diff1": 1.0, "naucs_at_20_max": 1.0, "naucs_at_20_std": 1.0, "naucs_at_20_diff1": 1.0, "naucs_at_50_max": NaN, "naucs_at_50_std": NaN, "naucs_at_50_diff1": NaN, "naucs_at_100_max": NaN, "naucs_at_100_std": NaN, "naucs_at_100_diff1": NaN}}

special_tokens_map.json

@@ -0,0 +1,53 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<fake_token_around_image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<end_of_utterance>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": {
+    "content": "<|im_start|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,1183 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<repo_name>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<reponame>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<file_sep>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<filename>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<gh_stars>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<jupyter_script>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49152": {
+      "content": "<global-img>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49153": {
+      "content": "<row_1_col_1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49154": {
+      "content": "<row_1_col_2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49155": {
+      "content": "<row_1_col_3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49156": {
+      "content": "<row_1_col_4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49157": {
+      "content": "<row_1_col_5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49158": {
+      "content": "<row_1_col_6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49159": {
+      "content": "<row_2_col_1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49160": {
+      "content": "<row_2_col_2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49161": {
+      "content": "<row_2_col_3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49162": {
+      "content": "<row_2_col_4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49163": {
+      "content": "<row_2_col_5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49164": {
+      "content": "<row_2_col_6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49165": {
+      "content": "<row_3_col_1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49166": {
+      "content": "<row_3_col_2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49167": {
+      "content": "<row_3_col_3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49168": {
+      "content": "<row_3_col_4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49169": {
+      "content": "<row_3_col_5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49170": {
+      "content": "<row_3_col_6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49171": {
+      "content": "<row_4_col_1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49172": {
+      "content": "<row_4_col_2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49173": {
+      "content": "<row_4_col_3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49174": {
+      "content": "<row_4_col_4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49175": {
+      "content": "<row_4_col_5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49176": {
+      "content": "<row_4_col_6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49177": {
+      "content": "<row_5_col_1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49178": {
+      "content": "<row_5_col_2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49179": {
+      "content": "<row_5_col_3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49180": {
+      "content": "<row_5_col_4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49181": {
+      "content": "<row_5_col_5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49182": {
+      "content": "<row_5_col_6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49183": {
+      "content": "<row_6_col_1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49184": {
+      "content": "<row_6_col_2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49185": {
+      "content": "<row_6_col_3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49186": {
+      "content": "<row_6_col_4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49187": {
+      "content": "<row_6_col_5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49188": {
+      "content": "<row_6_col_6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49189": {
+      "content": "<fake_token_around_image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49190": {
+      "content": "<image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49191": {
+      "content": "<|reserved_special_token_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49192": {
+      "content": "<|reserved_special_token_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49193": {
+      "content": "<|reserved_special_token_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49194": {
+      "content": "<|reserved_special_token_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49195": {
+      "content": "<|reserved_special_token_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49196": {
+      "content": "<|reserved_special_token_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49197": {
+      "content": "<|reserved_special_token_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49198": {
+      "content": "<|reserved_special_token_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49199": {
+      "content": "<|reserved_special_token_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49200": {
+      "content": "<|reserved_special_token_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49201": {
+      "content": "<|reserved_special_token_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49202": {
+      "content": "<|reserved_special_token_11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49203": {
+      "content": "<|reserved_special_token_12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49204": {
+      "content": "<|reserved_special_token_13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49205": {
+      "content": "<|reserved_special_token_14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49206": {
+      "content": "<|reserved_special_token_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49207": {
+      "content": "<|reserved_special_token_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49208": {
+      "content": "<|reserved_special_token_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49209": {
+      "content": "<|reserved_special_token_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49210": {
+      "content": "<|reserved_special_token_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49211": {
+      "content": "<|reserved_special_token_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49212": {
+      "content": "<|reserved_special_token_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49213": {
+      "content": "<|reserved_special_token_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49214": {
+      "content": "<|reserved_special_token_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49215": {
+      "content": "<|reserved_special_token_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49216": {
+      "content": "<|reserved_special_token_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49217": {
+      "content": "<|reserved_special_token_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49218": {
+      "content": "<|reserved_special_token_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49219": {
+      "content": "<|reserved_special_token_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49220": {
+      "content": "<|reserved_special_token_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49221": {
+      "content": "<|reserved_special_token_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49222": {
+      "content": "<|reserved_special_token_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49223": {
+      "content": "<|reserved_special_token_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49224": {
+      "content": "<|reserved_special_token_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49225": {
+      "content": "<|reserved_special_token_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49226": {
+      "content": "<|reserved_special_token_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49227": {
+      "content": "<|reserved_special_token_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49228": {
+      "content": "<|reserved_special_token_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49229": {
+      "content": "<|reserved_special_token_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49230": {
+      "content": "<|reserved_special_token_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49231": {
+      "content": "<|reserved_special_token_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49232": {
+      "content": "<|reserved_special_token_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49233": {
+      "content": "<|reserved_special_token_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49234": {
+      "content": "<|reserved_special_token_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49235": {
+      "content": "<|reserved_special_token_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49236": {
+      "content": "<|reserved_special_token_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49237": {
+      "content": "<|reserved_special_token_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49238": {
+      "content": "<|reserved_special_token_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49239": {
+      "content": "<|reserved_special_token_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49240": {
+      "content": "<|reserved_special_token_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49241": {
+      "content": "<|reserved_special_token_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49242": {
+      "content": "<|reserved_special_token_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49243": {
+      "content": "<|reserved_special_token_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49244": {
+      "content": "<|reserved_special_token_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49245": {
+      "content": "<|reserved_special_token_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49246": {
+      "content": "<|reserved_special_token_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49247": {
+      "content": "<|reserved_special_token_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49248": {
+      "content": "<|reserved_special_token_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49249": {
+      "content": "<|reserved_special_token_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49250": {
+      "content": "<|reserved_special_token_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49251": {
+      "content": "<|reserved_special_token_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49252": {
+      "content": "<|reserved_special_token_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49253": {
+      "content": "<|reserved_special_token_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49254": {
+      "content": "<|reserved_special_token_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49255": {
+      "content": "<|reserved_special_token_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49256": {
+      "content": "<|reserved_special_token_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49257": {
+      "content": "<|reserved_special_token_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49258": {
+      "content": "<|reserved_special_token_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49259": {
+      "content": "<|reserved_special_token_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49260": {
+      "content": "<|reserved_special_token_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49261": {
+      "content": "<|reserved_special_token_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49262": {
+      "content": "<|reserved_special_token_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49263": {
+      "content": "<|reserved_special_token_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49264": {
+      "content": "<|reserved_special_token_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49265": {
+      "content": "<|reserved_special_token_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49266": {
+      "content": "<|reserved_special_token_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49267": {
+      "content": "<|reserved_special_token_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49268": {
+      "content": "<|reserved_special_token_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49269": {
+      "content": "<|reserved_special_token_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49270": {
+      "content": "<|reserved_special_token_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49271": {
+      "content": "<|reserved_special_token_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49272": {
+      "content": "<|reserved_special_token_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49273": {
+      "content": "<|reserved_special_token_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49274": {
+      "content": "<|reserved_special_token_83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49275": {
+      "content": "<|reserved_special_token_84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49276": {
+      "content": "<|reserved_special_token_85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49277": {
+      "content": "<|reserved_special_token_86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49278": {
+      "content": "<|reserved_special_token_87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49279": {
+      "content": "<end_of_utterance>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<fake_token_around_image>",
+    "<image>",
+    "<end_of_utterance>"
+  ],
+  "bos_token": "<|im_start|>",
+  "chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": {},
+  "legacy": false,
+  "model_max_length": 8192,
+  "pad_token": "<|im_end|>",
+  "processor_class": "ColIdefics3Processor",
+  "tokenizer_class": "GPT2Tokenizer",
+  "truncation_side": "left",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}

training_config.yml

@@ -0,0 +1,72 @@
+config:
+  (): colpali_engine.trainer.colmodel_training.ColModelTrainingConfig
+  output_dir: !path ../../../models/ColSmolVLM-Instruct-500M
+  processor:
+    (): colpali_engine.utils.transformers_wrappers.AllPurposeWrapper
+    class_to_instanciate: !ext colpali_engine.models.ColIdefics3Processor
+    pretrained_model_name_or_path:  "./models/ColSmolVLM-Instruct-500M"
+    # num_image_tokens: 2048
+    # max_length: 50
+
+  model:
+    (): colpali_engine.utils.transformers_wrappers.AllPurposeWrapper
+    class_to_instanciate: !ext colpali_engine.models.ColIdefics3
+    pretrained_model_name_or_path: "./models/ColSmolVLM-Instruct-500M"
+    torch_dtype:  !ext torch.bfloat16
+    # use_cache: false
+    attn_implementation: "flash_attention_2"
+#    device_map: "auto"
+#    quantization_config:
+#      (): transformers.BitsAndBytesConfig
+#      load_in_4bit: true
+#      bnb_4bit_quant_type: "nf4"
+#      bnb_4bit_compute_dtype:  "bfloat16"
+#      bnb_4bit_use_double_quant: true
+
+  dataset_loading_func: !ext colpali_engine.utils.dataset_transformation.load_train_set
+  eval_dataset_loader: !import ../data/test_data.yaml
+
+  # max_length: 50
+  run_eval: true
+  loss_func:
+    (): colpali_engine.loss.late_interaction_losses.ColbertPairwiseCELoss
+  tr_args:
+    (): transformers.training_args.TrainingArguments
+    output_dir: null
+    overwrite_output_dir: true
+    num_train_epochs: 3
+    per_device_train_batch_size: 8
+    gradient_checkpointing: true
+    gradient_checkpointing_kwargs: { "use_reentrant": false }
+    # gradient_checkpointing: true
+    # 6 x 8 gpus = 48 batch size
+    # gradient_accumulation_steps: 4
+    per_device_eval_batch_size: 4
+    eval_strategy: "steps"
+    dataloader_num_workers: 6
+    # bf16: true
+    save_steps: 500
+    logging_steps: 10
+    eval_steps: 100
+    warmup_steps: 100
+    learning_rate: 5e-4
+    save_total_limit: 1
+    resume_from_checkpoint: true
+    # optim: "paged_adamw_8bit"
+    # wandb logging
+    # wandb_project: "colqwen2"
+    # run_name: "colqwen2-ba32-nolora"
+    report_to: "wandb"
+
+
+  peft_config:
+    (): peft.LoraConfig
+    r: 32
+    lora_alpha: 32
+    lora_dropout: 0.1
+    init_lora_weights: "gaussian"
+    bias: "none"
+    task_type: "FEATURE_EXTRACTION"
+    target_modules: '(.*(model.text_model).*(down_proj|gate_proj|up_proj|k_proj|q_proj|v_proj|o_proj).*$|.*(custom_text_proj).*$)'
+    # target_modules: '(.*(language_model).*(down_proj|gate_proj|up_proj|k_proj|q_proj|v_proj|o_proj).*$|.*(custom_text_proj).*$)'
+