Upload folder using huggingface_hub

Browse files

Files changed (13) hide show

.gitattributes +1 -0
README.md +220 -36
added_tokens.json +32 -0
chat_template.jinja +17 -0
config.json +44 -0
generation_config.json +8 -0
merges.txt +0 -0
model.safetensors +3 -0
quantization_config.json +0 -0
special_tokens_map.json +46 -0
tokenizer.json +3 -0
tokenizer_config.json +263 -0
vocab.json +0 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md CHANGED Viewed

@@ -1,39 +1,223 @@
 ---
-base_model: qingy2024/GRMR-V3-Q4B
-base_model_relation: quantized
-quantized_by: ArtusDev
 ---
-## EXL3 Quants of qingy2024/GRMR-V3-Q4B
-EXL3 quants of [qingy2024/GRMR-V3-Q4B](https://huggingface.co/qingy2024/GRMR-V3-Q4B) using <a href="https://github.com/turboderp-org/exllamav3/">exllamav3</a> for quantization.
-### Quants
-| Quant(Revision) | Bits per Weight | Head Bits |
-| -------- | ---------- | --------- |
-| [3.0_H6](https://huggingface.co/ArtusDev/qingy2024_GRMR-V3-Q4B-EXL3/tree/3.0bpw_H6) | 3.0 | 6 |
-| [3.5_H6](https://huggingface.co/ArtusDev/qingy2024_GRMR-V3-Q4B-EXL3/tree/3.5bpw_H6) | 3.5 | 6 |
-| [4.0_H6](https://huggingface.co/ArtusDev/qingy2024_GRMR-V3-Q4B-EXL3/tree/4.0bpw_H6) | 4.0 | 6 |
-| [4.5_H6](https://huggingface.co/ArtusDev/qingy2024_GRMR-V3-Q4B-EXL3/tree/4.5bpw_H6) | 4.5 | 6 |
-| [5.0_H6](https://huggingface.co/ArtusDev/qingy2024_GRMR-V3-Q4B-EXL3/tree/5.0bpw_H6) | 5.0 | 6 |
-| [6.0_H6](https://huggingface.co/ArtusDev/qingy2024_GRMR-V3-Q4B-EXL3/tree/6.0bpw_H6) | 6.0 | 6 |
-| [8.0_H6](https://huggingface.co/ArtusDev/qingy2024_GRMR-V3-Q4B-EXL3/tree/8.0bpw_H6) | 8.0 | 6 |
-| [8.0_H8](https://huggingface.co/ArtusDev/qingy2024_GRMR-V3-Q4B-EXL3/tree/8.0bpw_H8) | 8.0 | 8 |
-### Downloading quants with huggingface-cli
-<details>
-  <summary>Click to view download instructions</summary>
-Install hugginface-cli:
-```bash
-pip install -U "huggingface_hub[cli]"
-```
-Download quant by targeting the specific quant revision (branch):
-```
-huggingface-cli download ArtusDev/qingy2024_GRMR-V3-Q4B-EXL3 --revision "5bpw_H6" --local-dir ./
-```
-</details>

 ---
+base_model: unsloth/Qwen3-4B-Base
+tags:
+- text-generation-inference
+- transformers
+- unsloth
+- qwen
+- trl
+- sft
+license: apache-2.0
+language:
+- en
 ---
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+</head>
+<div class="container"><h1>GRMR-V3-Q4B</h1><p>GRMR-V3-Q4B is a fine-tuned version of <a href="https://huggingface.co/unsloth/Qwen3-4B-Base">unsloth/Qwen3-4B-Base</a> specifically optimized for grammar correction tasks.</p><div class="important-note"><p><strong>IMPORTANT:</strong> Please ensure you are using the following sampler settings for optimal results:</p><pre><code>temperature = 0.7
+frequency_penalty = 0.0
+presence_penalty = 0.0
+min_p = 0.01
+top_p = 0.95
+top_k = 40</code></pre></div><h2>Model description</h2><p>GRMR-V3-Q4B is a grammar correction model built on Qwen3 4B base model. It has been fine-tuned on a large dataset of grammar correction examples to help improve text quality by fixing grammatical errors, punctuation, spelling, and other language issues.</p><p>The model uses a specialized chat template that structures inputs as "text" and outputs as "corrected" to maintain a clear distinction between original and corrected content.</p><h3>Model Details</h3><div class="model-meta"><p><strong>Developed by:</strong> qingy2024</p><p><strong>Finetuned from model:</strong> <a href="https://huggingface.co/unsloth/Qwen3-4B-Base">unsloth/Qwen3-4B-Base</a></p><p><strong>Model type:</strong> Qwen3</p><p><strong>Language(s):</strong> English</p><p><strong>License:</strong> apache-2.0</p><p><strong>Training Dataset:</strong> <a href="https://huggingface.co/datasets/qingy2024/grmr-v4-60k">qingy2024/grmr-v4-60k</a></p></div><hr><h2>Manual Testing Examples</h2><p>Here are a few examples of grammar corrections this model can handle:</p><table><thead><tr><th>Original Text</th><th>Corrected Text</th></tr></thead><tbody><tr><td>i dont know weather to bring a umbrella today</td><td>I don't know whether to bring an umbrella today.</td></tr><tr><td>she go to the store yesterday</td><td>She went to the store yesterday.</td></tr><tr><td>they is going to be late for the meeting</td><td>They are going to be late for the meeting.</td></tr><tr><td>the cat laying on the floor all day</td><td>The cat is laying on the floor all day.</td></tr></tbody></table><hr><h2>Training procedure</h2><p>The model was fine-tuned using full parameter fine-tuning (not LoRA) on the GRMR-V4-60K dataset. The training was optimized using the Unsloth framework for efficient training of LLMs.</p><h3>Training hyperparameters</h3><ul><li><strong>Batch size:</strong> 8</li><li><strong>Gradient accumulation steps:</strong> 2</li><li><strong>Learning rate:</strong> 5e-5</li><li><strong>Epochs:</strong> 1</li><li><strong>Optimizer:</strong> AdamW (8-bit)</li><li><strong>Weight decay:</strong> 0.01</li><li><strong>LR scheduler:</strong> Cosine</li><li><strong>Warmup steps:</strong> 180</li><li><strong>Max sequence length:</strong> 16,384</li><li><strong>Training precision:</strong> Mixed precision (BF16 where available, FP16 otherwise)</li></ul><h2>Intended uses & limitations</h2><p>This model is designed for grammar correction tasks. It can be used to:</p><ul><li>Fix grammatical errors in written text</li><li>Correct punctuation</li><li>Address spelling mistakes</li><li>Improve sentence structure and clarity</li></ul><h3>Limitations</h3><ul><li>The model may struggle with highly technical or domain-specific content</li><li>It may not fully understand context-dependent grammar rules in all cases</li><li>Performance may vary for non-standard English or text with multiple errors</li></ul><h2>How to use</h2><p>Projects based on Hugging Face transformers should be able to run this model easily.</p><pre><code class="language-python">from transformers import AutoModelForCausalLM, AutoTokenizer
+# Load model and tokenizer
+model_name = "qingy2024/GRMR-V3-Q4B"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name)
+# Text with grammar errors to correct
+text_to_correct = "i am going to the store tommorow and buy some thing for dinner"
+# Format as messages
+messages = [
+    {"role": "user", "content": text_to_correct}
+]
+# Apply the custom chat template
+prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+# Tokenize and generate
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(
+    inputs["input_ids"],
+    max_new_tokens=512,
+    temperature=0.1,
+    do_sample=True
+)
+# Decode and print the corrected text
+corrected_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(corrected_text)</code></pre><h3>Using with the Hugging Face pipeline</h3><pre><code class="language-python">from transformers import pipeline
+pipe = pipeline(
+    "text-generation",
+    model="qingy2024/GRMR-V3-Q4B",
+    torch_dtype="auto",
+    device_map="auto"
+)
+messages = [
+    {"role": "user", "content": "i dont know weather to bring a umbrella today"}
+]
+result = pipe(
+    messages,
+    max_new_tokens=100,
+    temperature=0.1,
+    do_sample=True,
+    return_full_text=False
+)[0]["generated_text"]
+print(result)</code></pre><h2>Custom Chat Template</h2><p class="chat-template-info">The model uses a custom chat template with special formatting for grammar correction:</p><ul><li>User inputs are formatted with <code><|text_start|></code> and <code><|text_end|></code> tags</li><li>Model outputs are formatted with <code><|corrected_start|></code> and <code><|corrected_end|></code> tags</li></ul><p>The complete chat template is:</p><pre><code class="language-jinja">{%- for message in messages %}
+{%- if message.role == "user" %}
+{{- '<|text_start|>\n' + message.content + '<|text_end|>\n' }}
+{%- elif message.role == "assistant" %}
+{{- '<|corrected_start|>\n' + message.content + '<|corrected_end|>\n' }}
+{%- else %}
+{{- raise('Unknown role: ' + message.role) }}
+{%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+{{- '<|corrected_start|>\n' }}
+{%- endif %}</code></pre><h2>Training Dataset</h2><p>The model was fine-tuned on the <a href="https://huggingface.co/datasets/qingy2024/grmr-v4-60k">qingy2024/grmr-v4-60k</a> dataset, which contains 60,000 examples of original text and their grammatically corrected versions.</p><h2>Bias, Risks, and Limitations</h2><ul><li>The model may reflect biases present in the training data</li><li>It may not perform equally well across different writing styles or domains</li><li>The model might occasionally introduce errors or change the meaning of text</li><li>It focuses on grammatical correctness rather than stylistic improvements</li></ul><h2>Citations</h2><pre><code>@misc{qwen3technicalreport,
+      title={Qwen3 Technical Report},
+      author={Qwen Team},
+      year={2025},
+      eprint={2505.09388},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2505.09388},
+}</code></pre><h2>Contact</h2><p>For questions or issues related to the model, please reach out via Hugging Face or by creating an issue in the repository.</p></div>
+<style>
+body {
+    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol";
+    line-height: 1.6;
+    margin: 0;
+    padding: 0;
+    background-color: #f8f9fa;
+    color: #333;
+}
+.container {
+    max-width: 1200px;
+    margin: 10px auto;
+    padding: 25px;
+    background-color: #ffffff;
+    border-radius: 8px;
+    box-shadow: 0 4px 12px rgba(0, 0, 0, 0.08);
+}
+h1, h2, h3 {
+    color: #0056b3; /* Primary Blue */
+    margin-top: 1.5em;
+    margin-bottom: 0.7em;
+}
+h1 {
+    text-align: center;
+    font-size: 2.2em;
+    border-bottom: 2px solid #e0e0e0;
+    padding-bottom: 0.5em;
+    margin-top: 0;
+}
+h2 {
+    font-size: 1.8em;
+    border-bottom: 1px solid #e9ecef;
+    padding-bottom: 0.3em;
+}
+h3 {
+    font-size: 1.4em;
+    color: #007bff; /* Lighter Blue for sub-headings */
+}
+p, li {
+    font-size: 1em;
+    color: #555;
+}
+a {
+    color: #007bff;
+    text-decoration: none;
+}
+a:hover {
+    text-decoration: underline;
+    color: #0056b3;
+}
+.important-note {
+    background-color: #e7f3ff; /* Light blue background */
+    border-left: 5px solid #007bff; /* Blue accent border */
+    margin: 20px 0px;
+    border-radius: 5px;
+}
+.important-note strong {
+    color: #0056b3;
+    font-weight: 600;
+}
+.important-note {
+    background-color: #d0e8ff;
+    padding: 0.05em 1.0em;
+    border-radius: 3px;
+    font-size: 0.9em;
+}
+code {
+    padding: 0.1em 0.4em;
+    border-radius: 3px;
+    font-size: 0.9em;
+}
+table {
+    width: 100%;
+    border-collapse: collapse;
+    margin: 20px 0;
+    box-shadow: 0 2px 4px rgba(0,0,0,0.05);
+}
+th, td {
+    border: 1px solid #dee2e6;
+    padding: 10px 12px;
+    text-align: left;
+    vertical-align: top;
+}
+th {
+    background-color: #e9ecef; /* Light gray for headers */
+    font-weight: 600;
+    color: #212529;
+}
+td:first-child {
+    /* font-style: italic; */
+    color: #444;
+}
+pre {
+    background-color: #f1f3f5;
+    padding: 15px;
+    border-radius: 5px;
+    overflow-x: auto;
+    border: 1px solid #ced4da;
+    font-size: 0.9em;
+}
+code {
+    font-family: "SFMono-Regular", Consolas, "Liberation Mono", Menlo, Courier, monospace;
+    background-color: #e9ecef;
+    padding: 0.2em 0.4em;
+    border-radius: 3px;
+    font-size: 0.9em;
+}
+pre code {
+    background-color: transparent;
+    padding: 0;
+    border-radius: 0;
+    font-size: 1em;
+}
+ul {
+    padding-left: 20px;
+}
+li {
+    margin-bottom: 0.5em;
+}
+hr {
+    border: none;
+    border-top: 1px solid #e0e0e0;
+    margin: 30px 0;
+}
+.model-meta {
+    background-color: #f8f9fa;
+    padding: 15px;
+    border-radius: 5px;
+    margin-bottom: 20px;
+    border: 1px solid #e9ecef;
+}
+.model-meta p { margin-bottom: 0.5em; }
+.model-meta strong { color: #333; }
+/* Specific styling for chat template explanation */
+.chat-template-info span {
+    font-weight: bold;
+    color: #0056b3;
+}
+</style></html>

added_tokens.json ADDED Viewed

	@@ -0,0 +1,32 @@

+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|corrected_end|>": 151672,
+  "<|corrected_start|>": 151671,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|text_end|>": 151670,
+  "<|text_start|>": 151669,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.jinja ADDED Viewed

	@@ -0,0 +1,17 @@

+{%- for message in messages %}
+{%- if message.role == "user" %}
+{{- '<|text_start|>
+' + message.content + '<|text_end|>
+' }}
+{%- elif message.role == "assistant" %}
+{{- '<|corrected_start|>
+' + message.content + '<|corrected_end|>
+' }}
+{%- else %}
+{{- raise('Unknown role: ' + message.role) }}
+{%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+{{- '<|corrected_start|>
+' }}
+{%- endif %}

config.json ADDED Viewed

	@@ -0,0 +1,44 @@

+{
+    "architectures": [
+        "Qwen3ForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "chat_template": "{%- for message in messages %}\n{%- if message.role == \"user\" %}\n{{- '<|text_start|>\n' + message.content + '<|text_end|>\n' }}\n{%- elif message.role == \"assistant\" %}\n{{- '<|corrected_start|>\n' + message.content + '<|corrected_end|>\n' }}\n{%- else %}\n{# Raise an error for unsupported roles, as per requirement to remove system message stuff #}\n{{- raise('Unknown role: ' + message.role) }}\n{%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n{{- '<|corrected_start|>\n' }}\n{%- endif %}",
+    "eos_token_id": 151643,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 2560,
+    "initializer_range": 0.02,
+    "intermediate_size": 9728,
+    "max_position_embeddings": 32768,
+    "max_window_layers": 36,
+    "model_type": "qwen3",
+    "num_attention_heads": 32,
+    "num_hidden_layers": 36,
+    "num_key_value_heads": 8,
+    "pad_token_id": 151654,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": null,
+    "rope_theta": 1000000,
+    "sliding_window": null,
+    "tie_word_embeddings": true,
+    "torch_dtype": "bfloat16",
+    "transformers_version": "4.52.4",
+    "unsloth_fixed": true,
+    "unsloth_version": "2025.5.9",
+    "use_cache": true,
+    "use_sliding_window": false,
+    "vocab_size": 151673,
+    "quantization_config": {
+        "quant_method": "exl3",
+        "version": "0.0.3",
+        "bits": 4.5,
+        "head_bits": 6,
+        "calibration": {
+            "rows": 100,
+            "cols": 2048
+        },
+        "out_scales": "auto"
+    }
+}

generation_config.json ADDED Viewed

	@@ -0,0 +1,8 @@

+{
+  "bos_token_id": 151643,
+  "eos_token_id": 151643,
+  "max_length": 32768,
+  "max_new_tokens": 2048,
+  "pad_token_id": 151654,
+  "transformers_version": "4.52.4"
+}

merges.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:20b69cca0304a35581fa3d14140cf725a11cae54ccee4f6e258b82ac222f402b
+size 3116132144

quantization_config.json ADDED Viewed

The diff for this file is too large to render. See raw diff

special_tokens_map.json ADDED Viewed

	@@ -0,0 +1,46 @@

+{
+  "additional_special_tokens": [
+    {
+      "content": "<|text_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|text_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|corrected_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|corrected_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|vision_pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:40fd5b853d0598451f720d22bf96c9c4cd4b5c6413cd74d4f5e3494ee990f38d
+size 11423523

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,263 @@

+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151669": {
+      "content": "<|text_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151670": {
+      "content": "<|text_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151671": {
+      "content": "<|corrected_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151672": {
+      "content": "<|corrected_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|text_start|>",
+    "<|text_end|>",
+    "<|corrected_start|>",
+    "<|corrected_end|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 32768,
+  "pad_token": "<|vision_pad|>",
+  "padding_side": "right",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

vocab.json ADDED Viewed

The diff for this file is too large to render. See raw diff