Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/intro_bar.png filter=lfs diff=lfs merge=lfs -text
+assets/method.png filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Agent-RL
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,171 @@
+---
+license: mit
+datasets:
+- RUC-NLPIR/FlashRAG_datasets
+base_model:
+- Qwen/Qwen2.5-32B
+---
+<div align="center">
+
+# ***ReSearch***: Learning to ***Re***ason with ***Search*** for LLMs via Reinforcement Learning
+
+[![Arxiv](https://img.shields.io/badge/paper-A82F27?style=for-the-badge&logo=arxiv)](https://arxiv.org/abs/2503.19470) 
+<!-- [![Model](https://img.shields.io/badge/model-4169E1?style=for-the-badge&logo=huggingface)](https://arxiv.org/abs/2503.19470)  -->
+
+</div>
+
+<p align="center">
+<img src="./assets/intro_bar.png" width="90%" alt="Intro" />
+<img src="./assets/method.png" width="90%" alt="Method" />
+</p>
+
+We propose ***ReSearch***, a novel framework that trains LLMs to ***Re***ason with ***Search*** via reinforcement learning without using any supervised data on reasoning steps. Our approach treats search operations as integral components of the reasoning chain, where when and how to perform searches is guided by text-based thinking, and search results subsequently influence further reasoning.
+
+## 📰 News
+- **[2025-03-26]** 🎉 We release the paper, update the code and open-source the models.
+  - 📝 The **paper is released** on arXiv, more details and evaluation results can be found in our [paper](https://arxiv.org/abs/2503.19470).
+  - 🛠️ The **repository is updated** with the new implementation, especially the rollout with search during RL training. This version of implementation is based on the latest release of verl.
+- **[2025-03-03]** ✅ We have released the preview version of ReSearch implementation.
+
+## 📦 Installation
+
+We recommend using conda to manage the environment. First create a conda environment and activate it.
+```bash
+conda create -n re-search python==3.10
+conda activate re-search
+```
+Then install dependencies, and our modified verl and flashrag packages  under ```src/``` will be installed in the editable mode.  Check out ```setup.py``` for details.
+```bash
+pip3 install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu124
+pip3 install flash-attn --no-build-isolation
+git clone https://github.com/Agent-RL/ReSearch.git
+cd ReSearch
+pip3 install -e .
+```
+As described in the [FlashRAG](https://github.com/RUC-NLPIR/FlashRAG?tab=readme-ov-file#wrench-installation), due to the incompatibility when installing faiss using pip, we need to use the following conda command to install faiss-gpu.
+```bash
+conda install -c pytorch -c nvidia faiss-gpu=1.8.0
+```
+
+## 🚀 Quick Start
+
+### Retriever Serving
+
+As described in our paper, during model training and evaluation, search operation will be conducted in the rollout and inference process. In practice, we host a retriever service via FlashRAG and FastAPI. Hence, the search operation is standardized to be an API call. This serving can be used to decouple the search operation from the reinforcement learning process, making the training and evaluation more clear and flexible.
+
+Before starting the retriever serving, you need download the [pre-indexed wikipedia](https://github.com/RUC-NLPIR/FlashRAG?tab=readme-ov-file#index), [wikipedia corpus and corresponding retriever models](https://github.com/RUC-NLPIR/FlashRAG/blob/main/docs/original_docs/reproduce_experiment.md#preliminary). More details can be found in the documentation of FlashRAG.
+
+For starting the retriever serving, you need to first fill the `scripts/serving/retriever_config.yaml` with the correct path to the retrieval model, index, and corpus, and available GPU ids. Then, you can run the following command to start the retriever serving:
+```bash
+cd scripts/serving
+python retriever_serving.py \
+    --config retriever_config.yaml \
+    --num_retriever {num_retriever} \  
+    --port {port}
+```
+
+The started retriever serving will be used in the training and evaluation process in the following part.
+
+### Data Preparation
+
+*ReSearch* is trained on the training set of MuSiQue, and evaluated on the dev set of HotpotQA, 2WikiMultiHopQA, MuSiQue and Bamboogle. For downloading the datasets, please refer to the `data/download_dataset.sh` script.
+```bash
+cd data
+bash download_dataset.sh
+```
+
+For preparing the training and validation data for following reinforcement learning, please run this script to parse the MuSiQue dataset to the parquet format.
+```bash
+cd data
+python prepare_musique.py
+```
+
+### Training
+
+Our training framework is based on [verl](https://github.com/volcengine/verl), a powerful reinforcement learning framework for LLMs. We deeply customize the verl code to fit our needs, and the modified version of verl is under the `src/verl` directory. The example of training scripts are under `scripts/train`.
+
+#### Single-node training
+Here is an example of training Qwen2.5-7B-Instruct with 4 GPUs locally. Note that the training script below **is just an example** for single-node training, using small batch size for quick start, and do not assure the training performance.
+```bash
+cd scripts/train
+bash train.sh \
+    --train_batch_size 8 \
+    --ppo_mini_batch_size 8 \
+    --apply_chat True \
+    --prompt_template_name re_search_template_sys \
+    --actor_model_path {model/path/to/qwen2.5-7b-instruct} \
+    --search_url {your-hosted-retriever-url} \
+    --project_name {wandb-project-name} \
+    --experiment_name {wandb-experiment-name} \
+    --nnodes 1 \
+    --n_gpus_per_node 4 \
+    --save_freq 5 \
+    --test_freq 5 \
+    --total_epochs 2 \
+    --wandb_api_key {your-wandb-api-key} \
+    --save_path {path/to/save} \
+    --train_files {path/to/train/parquet/data} \
+    --test_files {path/to/test/parquet/data}
+```
+- For training base (pre-trained) models, please use `--apply_chat False` and `--prompt_template_name re_search_template`
+- For training instruction-tuned models, please use `--apply_chat True` and `--prompt_template_name re_search_template_sys`
+
+#### Multi-node training
+
+If you want to **fully reproduce** the results in our paper, please refer to the multi-node training script in `scripts/train/train_multi_node.sh`, as well as the implementation details in our paper.
+
+### Evaluation
+
+We recommend using [SGLang](https://docs.sglang.ai/) to serve the trained model. You can download our open-sourced models or trained your own models to conduct the evaluation. Here is an example of launching the model serving:
+```bash
+python3 -m sglang.launch_server \
+        --served-model-name {trained/model/name} \
+        --model-path {trained/model/path} \
+        --tp 2 \
+        --context-length 8192 \
+        --enable-metrics \
+        --dtype bfloat16 \
+        --host 0.0.0.0 \
+        --port 80 \
+        --trust-remote-code \
+        --disable-overlap \
+        --disable-radix-cache
+```
+
+We use [FlashRAG](https://github.com/RUC-NLPIR/FlashRAG) as the standard evaluation environment. Here is an example of evaluating the performance of ReSearch-Qwen-7B-Instruct on Bamboogle test set.
+```bash
+cd scripts/evaluation
+python run_eval.py \
+    --config_path eval_config.yaml \
+    --method_name research \
+    --data_dir {root/path/to/evaluation/data} \
+    --dataset_name bamboogle \
+    --split test \
+    --save_dir {your-save-dir} \
+    --save_note research_qwen7b_ins
+    --sgl_remote_url {your-launched-sgl-url} \
+    --remote_retriever_url {your-hosted-retriever-url} \
+    --generator_model {your-local-model-path} \
+    --apply_chat True
+```
+
+For base model, please use `--apply_chat False` and for instruction-tuned model, please use `--apply_chat True`, for loading correct prompt template when conducting evaluation for *ReSearch* model. For more details about the configuration, please refer to the `scripts/evaluation/eval_config.yaml` file. 
+
+## 🤝 Acknowledge
+
+This training implementation is based on [verl](https://github.com/volcengine/verl) and the evaluation is based on [FlashRAG](https://github.com/RUC-NLPIR/FlashRAG). The serving of retriever is based on [FastAPI](https://github.com/fastapi/fastapi). The model serving is based on [SGLang](https://docs.sglang.ai/). *ReSearch* models are trained based on [Qwen2.5](https://qwenlm.github.io/blog/qwen2.5/). We sincerely appreciate their contributions to the open-source community.
+
+## 📚 Citation
+
+If you find this work useful, please cite it as follows:
+```bibtex
+@misc{chen2025research
+  title={ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning}, 
+  author={Mingyang Chen and Tianpeng Li and Haoze Sun and Yijie Zhou and Chenzheng Zhu and Haofen Wang and Jeff Z. Pan and Wen Zhang and Huajun Chen and Fan Yang and Zenan Zhou and Weipeng Chen},
+  year={2025},
+  eprint={2503.19470},
+  archivePrefix={arXiv},
+  primaryClass={cs.AI},
+  url={https://arxiv.org/abs/2503.19470}, 
+}
+```

added_tokens.json

@@ -0,0 +1,24 @@
+{
+  "</tool_call>": 151658,
+  "<tool_call>": 151657,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

config.json

@@ -0,0 +1,28 @@
+{
+  "architectures": [
+    "Qwen2ForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "eos_token_id": 151643,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 27648,
+  "max_position_embeddings": 131072,
+  "max_window_layers": 64,
+  "model_type": "qwen2",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 64,
+  "num_key_value_heads": 8,
+  "pad_token_id": 151643,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 1000000.0,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.46.0",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 152064
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": 151643,
+  "pad_token_id": 151643,
+  "transformers_version": "4.46.0"
+}

model-00001-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40ff27a28cee8c83ba419dc98d7b6e63152513a58bb4d91cb5ba9620f6ecf25b
+size 4781728512

model-00002-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5fb10ed7767133b55733900f1a20bb7f17f80d323eafbc9ffae9bdead4543d52
+size 4991389456

model-00003-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:becab6df5571c18d202b942faed095b5445f36ead32770368d7bde841c7ed3ac
+size 4897035224

model-00004-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9fb3095a30f1b6bc948d30d299c41acbbf70b3ce76d21d0732205b6e55c5e6e0
+size 4991336664

model-00005-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49748cc4fde934461a7ffcc1dba294db2a9f33198841ecfb33f1af637b4baacf
+size 4928562968

model-00006-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:429ff32afcb020916df3dd4b1d9d1a661496f286d73f55c68e2de6a3bfb41075
+size 4823577792

model-00007-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c7cd90b11820b42ced65eea1b6b932f4dcb5f691615322cf542e27297656f77
+size 4907576536

model-00008-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ea3b7fc112c371dd60678a0b6c0576cdb41bbe6124a0d11085224a812f72d012
+size 4152557760

model-00009-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:efc354599d4dd9bac0ce7dd8b6541f57d0d9633fcc15dca1b2a65019994c7473
+size 4912669560

model-00010-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:35b64f1e03b1d925f617abd8245a09aec62c02612a5a5a6e4e8f028519a946e7
+size 4917941336

model-00011-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd40d975a7f599289b2cd250635f607a537be753c394c8239f3a9cdb1dbd1084
+size 4718765568

model-00012-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8c962d6903e360630fbeb4c86c35b5ae8994c62e80d7de6bc985c81516d972b8
+size 4786858032

model-00013-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a166d8fc69d13afa61216fed0195d9600a16ce2e56bca3f87a051ad3c6b047c0
+size 4802741928

model-00014-of-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b780365ee452cca90e79cc23e35cb68ef04c6641a6a4dc6ab143f3ab41aa36e3
+size 2915100152

model.safetensors.index.json

@@ -0,0 +1,778 @@
+{
+  "metadata": {
+    "total_size": 65527752704
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00012-of-00014.safetensors",
+    "model.embed_tokens.weight": "model-00009-of-00014.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00013-of-00014.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00007-of-00014.safetensors",
+    "model.layers.0.self_attn.k_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.0.self_attn.q_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.0.self_attn.v_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00013-of-00014.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00002-of-00014.safetensors",
+    "model.layers.1.self_attn.k_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.1.self_attn.q_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.1.self_attn.v_proj.bias": "model-00010-of-00014.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00011-of-00014.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00013-of-00014.safetensors",
+    "model.layers.10.self_attn.k_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.10.self_attn.q_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.10.self_attn.v_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00011-of-00014.safetensors",
+    "model.layers.11.self_attn.k_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.11.self_attn.q_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.11.self_attn.v_proj.bias": "model-00004-of-00014.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00001-of-00014.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00010-of-00014.safetensors",
+    "model.layers.12.self_attn.k_proj.bias": "model-00004-of-00014.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.12.self_attn.q_proj.bias": "model-00014-of-00014.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.12.self_attn.v_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00006-of-00014.safetensors",
+    "model.layers.13.self_attn.k_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.13.self_attn.q_proj.bias": "model-00010-of-00014.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.13.self_attn.v_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00003-of-00014.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00014.safetensors",
+    "model.layers.14.self_attn.k_proj.bias": "model-00009-of-00014.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.14.self_attn.q_proj.bias": "model-00004-of-00014.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.14.self_attn.v_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00013-of-00014.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00008-of-00014.safetensors",
+    "model.layers.15.self_attn.k_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.15.self_attn.q_proj.bias": "model-00009-of-00014.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.15.self_attn.v_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00003-of-00014.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.16.self_attn.k_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.16.self_attn.q_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.16.self_attn.v_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00002-of-00014.safetensors",
+    "model.layers.17.self_attn.k_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.17.self_attn.q_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.17.self_attn.v_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00013-of-00014.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00003-of-00014.safetensors",
+    "model.layers.18.self_attn.k_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.18.self_attn.q_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.18.self_attn.v_proj.bias": "model-00002-of-00014.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00013-of-00014.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00013-of-00014.safetensors",
+    "model.layers.19.self_attn.k_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.19.self_attn.q_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.19.self_attn.v_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00014.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00007-of-00014.safetensors",
+    "model.layers.2.self_attn.k_proj.bias": "model-00014-of-00014.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.2.self_attn.q_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.2.self_attn.v_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00002-of-00014.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00008-of-00014.safetensors",
+    "model.layers.20.self_attn.k_proj.bias": "model-00010-of-00014.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.20.self_attn.q_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.20.self_attn.v_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00009-of-00014.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00007-of-00014.safetensors",
+    "model.layers.21.self_attn.k_proj.bias": "model-00004-of-00014.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.21.self_attn.q_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.21.self_attn.v_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00006-of-00014.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.22.self_attn.k_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.22.self_attn.q_proj.bias": "model-00002-of-00014.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.22.self_attn.v_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00010-of-00014.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.23.self_attn.k_proj.bias": "model-00010-of-00014.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.23.self_attn.q_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.23.self_attn.v_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00003-of-00014.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00010-of-00014.safetensors",
+    "model.layers.24.self_attn.k_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.24.self_attn.q_proj.bias": "model-00002-of-00014.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.24.self_attn.v_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00011-of-00014.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.25.self_attn.k_proj.bias": "model-00012-of-00014.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.25.self_attn.q_proj.bias": "model-00010-of-00014.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.25.self_attn.v_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00011-of-00014.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00013-of-00014.safetensors",
+    "model.layers.26.self_attn.k_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.26.self_attn.q_proj.bias": "model-00012-of-00014.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.26.self_attn.v_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00007-of-00014.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00009-of-00014.safetensors",
+    "model.layers.27.self_attn.k_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.27.self_attn.q_proj.bias": "model-00002-of-00014.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.27.self_attn.v_proj.bias": "model-00006-of-00014.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00011-of-00014.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.28.self_attn.k_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.28.self_attn.q_proj.bias": "model-00006-of-00014.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.28.self_attn.v_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00013-of-00014.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00011-of-00014.safetensors",
+    "model.layers.29.self_attn.k_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.29.self_attn.q_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.29.self_attn.v_proj.bias": "model-00009-of-00014.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00012-of-00014.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00007-of-00014.safetensors",
+    "model.layers.3.self_attn.k_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.3.self_attn.q_proj.bias": "model-00014-of-00014.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.3.self_attn.v_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00014-of-00014.safetensors",
+    "model.layers.30.self_attn.k_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.30.self_attn.q_proj.bias": "model-00012-of-00014.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.30.self_attn.v_proj.bias": "model-00006-of-00014.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00006-of-00014.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00004-of-00014.safetensors",
+    "model.layers.31.self_attn.k_proj.bias": "model-00014-of-00014.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.31.self_attn.q_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.31.self_attn.v_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.32.input_layernorm.weight": "model-00001-of-00014.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.32.mlp.gate_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.32.mlp.up_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00002-of-00014.safetensors",
+    "model.layers.32.self_attn.k_proj.bias": "model-00014-of-00014.safetensors",
+    "model.layers.32.self_attn.k_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.32.self_attn.q_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.32.self_attn.q_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.32.self_attn.v_proj.bias": "model-00006-of-00014.safetensors",
+    "model.layers.32.self_attn.v_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.33.input_layernorm.weight": "model-00013-of-00014.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.33.mlp.gate_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.33.mlp.up_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00008-of-00014.safetensors",
+    "model.layers.33.self_attn.k_proj.bias": "model-00012-of-00014.safetensors",
+    "model.layers.33.self_attn.k_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.33.self_attn.q_proj.bias": "model-00004-of-00014.safetensors",
+    "model.layers.33.self_attn.q_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.33.self_attn.v_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.33.self_attn.v_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.34.input_layernorm.weight": "model-00003-of-00014.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.34.mlp.gate_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.34.mlp.up_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00006-of-00014.safetensors",
+    "model.layers.34.self_attn.k_proj.bias": "model-00010-of-00014.safetensors",
+    "model.layers.34.self_attn.k_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.34.self_attn.q_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.34.self_attn.q_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.34.self_attn.v_proj.bias": "model-00006-of-00014.safetensors",
+    "model.layers.34.self_attn.v_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.35.input_layernorm.weight": "model-00001-of-00014.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.35.mlp.gate_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.35.mlp.up_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00007-of-00014.safetensors",
+    "model.layers.35.self_attn.k_proj.bias": "model-00004-of-00014.safetensors",
+    "model.layers.35.self_attn.k_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.35.self_attn.q_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.35.self_attn.q_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.35.self_attn.v_proj.bias": "model-00012-of-00014.safetensors",
+    "model.layers.35.self_attn.v_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.36.input_layernorm.weight": "model-00008-of-00014.safetensors",
+    "model.layers.36.mlp.down_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.36.mlp.gate_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.36.mlp.up_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.36.post_attention_layernorm.weight": "model-00008-of-00014.safetensors",
+    "model.layers.36.self_attn.k_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.36.self_attn.k_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.36.self_attn.o_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.36.self_attn.q_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.36.self_attn.q_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.36.self_attn.v_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.36.self_attn.v_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.37.input_layernorm.weight": "model-00004-of-00014.safetensors",
+    "model.layers.37.mlp.down_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.37.mlp.gate_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.37.mlp.up_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.37.post_attention_layernorm.weight": "model-00001-of-00014.safetensors",
+    "model.layers.37.self_attn.k_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.37.self_attn.k_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.37.self_attn.o_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.37.self_attn.q_proj.bias": "model-00004-of-00014.safetensors",
+    "model.layers.37.self_attn.q_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.37.self_attn.v_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.37.self_attn.v_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.38.input_layernorm.weight": "model-00002-of-00014.safetensors",
+    "model.layers.38.mlp.down_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.38.mlp.gate_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.38.mlp.up_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.38.post_attention_layernorm.weight": "model-00004-of-00014.safetensors",
+    "model.layers.38.self_attn.k_proj.bias": "model-00014-of-00014.safetensors",
+    "model.layers.38.self_attn.k_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.38.self_attn.o_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.38.self_attn.q_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.38.self_attn.q_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.38.self_attn.v_proj.bias": "model-00004-of-00014.safetensors",
+    "model.layers.38.self_attn.v_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.39.input_layernorm.weight": "model-00003-of-00014.safetensors",
+    "model.layers.39.mlp.down_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.39.mlp.gate_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.39.mlp.up_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.39.post_attention_layernorm.weight": "model-00007-of-00014.safetensors",
+    "model.layers.39.self_attn.k_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.39.self_attn.k_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.39.self_attn.o_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.39.self_attn.q_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.39.self_attn.q_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.39.self_attn.v_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.39.self_attn.v_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00011-of-00014.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00013-of-00014.safetensors",
+    "model.layers.4.self_attn.k_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.4.self_attn.q_proj.bias": "model-00006-of-00014.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.4.self_attn.v_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.40.input_layernorm.weight": "model-00008-of-00014.safetensors",
+    "model.layers.40.mlp.down_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.40.mlp.gate_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.40.mlp.up_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.40.post_attention_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.40.self_attn.k_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.40.self_attn.k_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.40.self_attn.o_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.40.self_attn.q_proj.bias": "model-00006-of-00014.safetensors",
+    "model.layers.40.self_attn.q_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.40.self_attn.v_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.40.self_attn.v_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.41.input_layernorm.weight": "model-00001-of-00014.safetensors",
+    "model.layers.41.mlp.down_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.41.mlp.gate_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.41.mlp.up_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.41.post_attention_layernorm.weight": "model-00013-of-00014.safetensors",
+    "model.layers.41.self_attn.k_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.41.self_attn.k_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.41.self_attn.o_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.41.self_attn.q_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.41.self_attn.q_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.41.self_attn.v_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.41.self_attn.v_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.42.input_layernorm.weight": "model-00008-of-00014.safetensors",
+    "model.layers.42.mlp.down_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.42.mlp.gate_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.42.mlp.up_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.42.post_attention_layernorm.weight": "model-00007-of-00014.safetensors",
+    "model.layers.42.self_attn.k_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.42.self_attn.k_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.42.self_attn.o_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.42.self_attn.q_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.42.self_attn.q_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.42.self_attn.v_proj.bias": "model-00009-of-00014.safetensors",
+    "model.layers.42.self_attn.v_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.43.input_layernorm.weight": "model-00012-of-00014.safetensors",
+    "model.layers.43.mlp.down_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.43.mlp.gate_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.43.mlp.up_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.43.post_attention_layernorm.weight": "model-00007-of-00014.safetensors",
+    "model.layers.43.self_attn.k_proj.bias": "model-00014-of-00014.safetensors",
+    "model.layers.43.self_attn.k_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.43.self_attn.o_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.43.self_attn.q_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.43.self_attn.q_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.43.self_attn.v_proj.bias": "model-00014-of-00014.safetensors",
+    "model.layers.43.self_attn.v_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.44.input_layernorm.weight": "model-00001-of-00014.safetensors",
+    "model.layers.44.mlp.down_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.44.mlp.gate_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.44.mlp.up_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.44.post_attention_layernorm.weight": "model-00007-of-00014.safetensors",
+    "model.layers.44.self_attn.k_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.44.self_attn.k_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.44.self_attn.o_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.44.self_attn.q_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.44.self_attn.q_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.44.self_attn.v_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.44.self_attn.v_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.45.input_layernorm.weight": "model-00001-of-00014.safetensors",
+    "model.layers.45.mlp.down_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.45.mlp.gate_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.45.mlp.up_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.45.post_attention_layernorm.weight": "model-00003-of-00014.safetensors",
+    "model.layers.45.self_attn.k_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.45.self_attn.k_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.45.self_attn.o_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.45.self_attn.q_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.45.self_attn.q_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.45.self_attn.v_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.45.self_attn.v_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.46.input_layernorm.weight": "model-00012-of-00014.safetensors",
+    "model.layers.46.mlp.down_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.46.mlp.gate_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.46.mlp.up_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.46.post_attention_layernorm.weight": "model-00012-of-00014.safetensors",
+    "model.layers.46.self_attn.k_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.46.self_attn.k_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.46.self_attn.o_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.46.self_attn.q_proj.bias": "model-00006-of-00014.safetensors",
+    "model.layers.46.self_attn.q_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.46.self_attn.v_proj.bias": "model-00012-of-00014.safetensors",
+    "model.layers.46.self_attn.v_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.47.input_layernorm.weight": "model-00002-of-00014.safetensors",
+    "model.layers.47.mlp.down_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.47.mlp.gate_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.47.mlp.up_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.47.post_attention_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.47.self_attn.k_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.47.self_attn.k_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.47.self_attn.o_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.47.self_attn.q_proj.bias": "model-00009-of-00014.safetensors",
+    "model.layers.47.self_attn.q_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.47.self_attn.v_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.47.self_attn.v_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.48.input_layernorm.weight": "model-00012-of-00014.safetensors",
+    "model.layers.48.mlp.down_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.48.mlp.gate_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.48.mlp.up_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.48.post_attention_layernorm.weight": "model-00003-of-00014.safetensors",
+    "model.layers.48.self_attn.k_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.48.self_attn.k_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.48.self_attn.o_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.48.self_attn.q_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.48.self_attn.q_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.48.self_attn.v_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.48.self_attn.v_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.49.input_layernorm.weight": "model-00002-of-00014.safetensors",
+    "model.layers.49.mlp.down_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.49.mlp.gate_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.49.mlp.up_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.49.post_attention_layernorm.weight": "model-00010-of-00014.safetensors",
+    "model.layers.49.self_attn.k_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.49.self_attn.k_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.49.self_attn.o_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.49.self_attn.q_proj.bias": "model-00010-of-00014.safetensors",
+    "model.layers.49.self_attn.q_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.49.self_attn.v_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.49.self_attn.v_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00014-of-00014.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00002-of-00014.safetensors",
+    "model.layers.5.self_attn.k_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.5.self_attn.q_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.5.self_attn.v_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.50.input_layernorm.weight": "model-00001-of-00014.safetensors",
+    "model.layers.50.mlp.down_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.50.mlp.gate_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.50.mlp.up_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.50.post_attention_layernorm.weight": "model-00010-of-00014.safetensors",
+    "model.layers.50.self_attn.k_proj.bias": "model-00002-of-00014.safetensors",
+    "model.layers.50.self_attn.k_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.50.self_attn.o_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.50.self_attn.q_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.50.self_attn.q_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.50.self_attn.v_proj.bias": "model-00010-of-00014.safetensors",
+    "model.layers.50.self_attn.v_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.51.input_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.51.mlp.down_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.51.mlp.gate_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.51.mlp.up_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.51.post_attention_layernorm.weight": "model-00001-of-00014.safetensors",
+    "model.layers.51.self_attn.k_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.51.self_attn.k_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.51.self_attn.o_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.51.self_attn.q_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.51.self_attn.q_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.51.self_attn.v_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.51.self_attn.v_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.52.input_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.52.mlp.down_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.52.mlp.gate_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.52.mlp.up_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.52.post_attention_layernorm.weight": "model-00001-of-00014.safetensors",
+    "model.layers.52.self_attn.k_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.52.self_attn.k_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.52.self_attn.o_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.52.self_attn.q_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.52.self_attn.q_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.52.self_attn.v_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.52.self_attn.v_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.53.input_layernorm.weight": "model-00008-of-00014.safetensors",
+    "model.layers.53.mlp.down_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.53.mlp.gate_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.53.mlp.up_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.53.post_attention_layernorm.weight": "model-00012-of-00014.safetensors",
+    "model.layers.53.self_attn.k_proj.bias": "model-00009-of-00014.safetensors",
+    "model.layers.53.self_attn.k_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.53.self_attn.o_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.53.self_attn.q_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.53.self_attn.q_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.53.self_attn.v_proj.bias": "model-00004-of-00014.safetensors",
+    "model.layers.53.self_attn.v_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.54.input_layernorm.weight": "model-00007-of-00014.safetensors",
+    "model.layers.54.mlp.down_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.54.mlp.gate_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.54.mlp.up_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.54.post_attention_layernorm.weight": "model-00006-of-00014.safetensors",
+    "model.layers.54.self_attn.k_proj.bias": "model-00004-of-00014.safetensors",
+    "model.layers.54.self_attn.k_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.54.self_attn.o_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.54.self_attn.q_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.54.self_attn.q_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.54.self_attn.v_proj.bias": "model-00012-of-00014.safetensors",
+    "model.layers.54.self_attn.v_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.55.input_layernorm.weight": "model-00004-of-00014.safetensors",
+    "model.layers.55.mlp.down_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.55.mlp.gate_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.55.mlp.up_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.55.post_attention_layernorm.weight": "model-00001-of-00014.safetensors",
+    "model.layers.55.self_attn.k_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.55.self_attn.k_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.55.self_attn.o_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.55.self_attn.q_proj.bias": "model-00009-of-00014.safetensors",
+    "model.layers.55.self_attn.q_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.55.self_attn.v_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.55.self_attn.v_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.56.input_layernorm.weight": "model-00010-of-00014.safetensors",
+    "model.layers.56.mlp.down_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.56.mlp.gate_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.56.mlp.up_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.56.post_attention_layernorm.weight": "model-00013-of-00014.safetensors",
+    "model.layers.56.self_attn.k_proj.bias": "model-00006-of-00014.safetensors",
+    "model.layers.56.self_attn.k_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.56.self_attn.o_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.56.self_attn.q_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.56.self_attn.q_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.56.self_attn.v_proj.bias": "model-00002-of-00014.safetensors",
+    "model.layers.56.self_attn.v_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.57.input_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.57.mlp.down_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.57.mlp.gate_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.57.mlp.up_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.57.post_attention_layernorm.weight": "model-00013-of-00014.safetensors",
+    "model.layers.57.self_attn.k_proj.bias": "model-00012-of-00014.safetensors",
+    "model.layers.57.self_attn.k_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.57.self_attn.o_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.57.self_attn.q_proj.bias": "model-00006-of-00014.safetensors",
+    "model.layers.57.self_attn.q_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.57.self_attn.v_proj.bias": "model-00010-of-00014.safetensors",
+    "model.layers.57.self_attn.v_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.58.input_layernorm.weight": "model-00003-of-00014.safetensors",
+    "model.layers.58.mlp.down_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.58.mlp.gate_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.58.mlp.up_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.58.post_attention_layernorm.weight": "model-00010-of-00014.safetensors",
+    "model.layers.58.self_attn.k_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.58.self_attn.k_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.58.self_attn.o_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.58.self_attn.q_proj.bias": "model-00002-of-00014.safetensors",
+    "model.layers.58.self_attn.q_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.58.self_attn.v_proj.bias": "model-00009-of-00014.safetensors",
+    "model.layers.58.self_attn.v_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.59.input_layernorm.weight": "model-00013-of-00014.safetensors",
+    "model.layers.59.mlp.down_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.59.mlp.gate_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.59.mlp.up_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.59.post_attention_layernorm.weight": "model-00001-of-00014.safetensors",
+    "model.layers.59.self_attn.k_proj.bias": "model-00009-of-00014.safetensors",
+    "model.layers.59.self_attn.k_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.59.self_attn.o_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.59.self_attn.q_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.59.self_attn.q_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.59.self_attn.v_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.59.self_attn.v_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00012-of-00014.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.6.self_attn.k_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.6.self_attn.q_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.6.self_attn.v_proj.bias": "model-00009-of-00014.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.60.input_layernorm.weight": "model-00007-of-00014.safetensors",
+    "model.layers.60.mlp.down_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.60.mlp.gate_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.60.mlp.up_proj.weight": "model-00009-of-00014.safetensors",
+    "model.layers.60.post_attention_layernorm.weight": "model-00003-of-00014.safetensors",
+    "model.layers.60.self_attn.k_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.60.self_attn.k_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.60.self_attn.o_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.60.self_attn.q_proj.bias": "model-00002-of-00014.safetensors",
+    "model.layers.60.self_attn.q_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.60.self_attn.v_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.60.self_attn.v_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.61.input_layernorm.weight": "model-00006-of-00014.safetensors",
+    "model.layers.61.mlp.down_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.61.mlp.gate_proj.weight": "model-00008-of-00014.safetensors",
+    "model.layers.61.mlp.up_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.61.post_attention_layernorm.weight": "model-00001-of-00014.safetensors",
+    "model.layers.61.self_attn.k_proj.bias": "model-00004-of-00014.safetensors",
+    "model.layers.61.self_attn.k_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.61.self_attn.o_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.61.self_attn.q_proj.bias": "model-00007-of-00014.safetensors",
+    "model.layers.61.self_attn.q_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.61.self_attn.v_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.61.self_attn.v_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.62.input_layernorm.weight": "model-00011-of-00014.safetensors",
+    "model.layers.62.mlp.down_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.62.mlp.gate_proj.weight": "model-00006-of-00014.safetensors",
+    "model.layers.62.mlp.up_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.62.post_attention_layernorm.weight": "model-00004-of-00014.safetensors",
+    "model.layers.62.self_attn.k_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.62.self_attn.k_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.62.self_attn.o_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.62.self_attn.q_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.62.self_attn.q_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.62.self_attn.v_proj.bias": "model-00014-of-00014.safetensors",
+    "model.layers.62.self_attn.v_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.63.input_layernorm.weight": "model-00009-of-00014.safetensors",
+    "model.layers.63.mlp.down_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.63.mlp.gate_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.63.mlp.up_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.63.post_attention_layernorm.weight": "model-00002-of-00014.safetensors",
+    "model.layers.63.self_attn.k_proj.bias": "model-00011-of-00014.safetensors",
+    "model.layers.63.self_attn.k_proj.weight": "model-00005-of-00014.safetensors",
+    "model.layers.63.self_attn.o_proj.weight": "model-00014-of-00014.safetensors",
+    "model.layers.63.self_attn.q_proj.bias": "model-00004-of-00014.safetensors",
+    "model.layers.63.self_attn.q_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.63.self_attn.v_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.63.self_attn.v_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00011-of-00014.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00004-of-00014.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00009-of-00014.safetensors",
+    "model.layers.7.self_attn.k_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.7.self_attn.q_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.7.self_attn.v_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00007-of-00014.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00002-of-00014.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00007-of-00014.safetensors",
+    "model.layers.8.self_attn.k_proj.bias": "model-00008-of-00014.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.8.self_attn.q_proj.bias": "model-00013-of-00014.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00014.safetensors",
+    "model.layers.8.self_attn.v_proj.bias": "model-00005-of-00014.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00003-of-00014.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00003-of-00014.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00010-of-00014.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00007-of-00014.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00005-of-00014.safetensors",
+    "model.layers.9.self_attn.k_proj.bias": "model-00002-of-00014.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00013-of-00014.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00012-of-00014.safetensors",
+    "model.layers.9.self_attn.q_proj.bias": "model-00003-of-00014.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00011-of-00014.safetensors",
+    "model.layers.9.self_attn.v_proj.bias": "model-00001-of-00014.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00004-of-00014.safetensors",
+    "model.norm.weight": "model-00005-of-00014.safetensors"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa
+size 11421896

tokenizer_config.json

@@ -0,0 +1,207 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- for message in messages %}\n    {%- if message.role == 'system' %}\n        {{- '<B_SYS>' + message.content}}\n    {%- elif message.role == 'user' %}\n        {{- '<C_Q>' + message.content}}\n    {%- elif message.role == 'assistant' %}\n        {{- '<C_A>' + message.content}}\n    {%- elif message.role == 'think'%}\n        {%- if loop.last %}\n            {{- '<inner_think>' + message.content}}\n        {%- endif %}\n    {%- else %}\n        {{- raise_exception('Invalid message role: ' + message.role) }}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {%- if messages[-1].role == 'user' %}\n        {{- '<inner_think>'}}\n    {%- elif messages[-1].role == 'think' %}\n        {{- '<C_A>'}}\n    {%- endif %}\n{%- endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}