Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,295 @@
+---
+tags:
+- unsloth
+base_model:
+- inclusionAI/Ling-1T
+license: mit
+pipeline_tag: text-generation
+library_name: transformers
+---
+> [!NOTE]
+>  Includes Unsloth **chat template fixes**! <br> For `llama.cpp`, use `--jinja`
+>
+
+<div>
+<p style="margin-top: 0;margin-bottom: 0;">
+    <em><a href="https://docs.unsloth.ai/basics/unsloth-dynamic-v2.0-gguf">Unsloth Dynamic 2.0</a> achieves superior accuracy & outperforms other leading quants.</em>
+  </p>
+  <div style="display: flex; gap: 5px; align-items: center; ">
+    <a href="https://github.com/unslothai/unsloth/">
+      <img src="https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png" width="133">
+    </a>
+    <a href="https://discord.gg/unsloth">
+      <img src="https://github.com/unslothai/unsloth/raw/main/images/Discord%20button.png" width="173">
+    </a>
+    <a href="https://docs.unsloth.ai/">
+      <img src="https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/images/documentation%20green%20button.png" width="143">
+    </a>
+  </div>
+</div>
+
+
+<p align="center">
+    <img src="https://mdn.alipayobjects.com/huamei_qa8qxu/afts/img/A*4QxcQrBlTiAAAAAAQXAAAAgAemJ7AQ/original" width="100"/>
+</p>
+
+<p align="center">🤗 <a href="https://huggingface.co/inclusionAI">Hugging Face</a>&nbsp;&nbsp; | &nbsp;&nbsp;🤖 <a href="https://modelscope.cn/organization/inclusionAI">ModelScope </a>&nbsp;&nbsp; | &nbsp;&nbsp;🐙 <a href="https://zenmux.ai/inclusionai/ling-1t?utm_source=hf_inclusionAI">Experience Now</a></p>
+
+
+## Introduction
+
+**Ling-1T** is the first flagship *non-thinking* model in the Ling 2.0 series, featuring **1 trillion total parameters** with **≈ 50 billion active parameters per token**.
+Built on the Ling 2.0 architecture, Ling-1T is designed to push the limits of *efficient reasoning* and *scalable cognition*.
+
+Pre-trained on **20 trillion+ high-quality, reasoning-dense tokens**, Ling-1T-base supports up to **128K context length** and adopts an **evolutionary chain-of-thought (Evo-CoT)** process across mid-training and post-training.
+This curriculum greatly enhances the model’s efficiency and reasoning depth, allowing Ling-1T to achieve **state-of-the-art performance** on multiple complex reasoning benchmarks—balancing **accuracy** and **efficiency**.
+
+
+### Flagship-Level Efficient Reasoning
+
+<p align="center">
+    <img src="https://mdn.alipayobjects.com/huamei_bcz3yt/afts/img/YiXwTb4Q_vsAAAAAT-AAAAgADkV7AQFr/original"/>
+<p>
+
+<p align="center">
+    <img src="https://mdn.alipayobjects.com/huamei_bcz3yt/afts/img/MEh7Q5FtzbAAAAAAUQAAAAgADkV7AQFr/original"/>
+<p>
+
+We comprehensively evaluated Ling-1T against leading flagship models, including both **open-source giants** (e.g., *DeepSeek-V3.1-Terminus*, *Kimi-K2-Instruct-0905*) and **closed-source APIs** (*GPT-5-main*, *Gemini-2.5-Pro*).
+Across code generation, software development, competition-level mathematics, professional math, and logical reasoning, Ling-1T consistently demonstrates **superior complex reasoning ability** and overall advantage.
+
+In the **AIME 25** benchmark, Ling-1T extends the **Pareto frontier** of reasoning accuracy vs. reasoning length, showcasing its strength in **“efficient thinking and precise reasoning.”**
+
+<p align="center">
+    <img src="https://mdn.alipayobjects.com/huamei_bcz3yt/afts/img/J8ciS5KbIrwAAAAAceAAAAgADkV7AQFr/original"/>
+<p>
+
+### Aesthetic Understanding and Front-End Generation
+
+Ling-1T excels in visual reasoning and front-end code generation tasks, combining deep semantic understanding with precise code synthesis.
+We introduce a hybrid *Syntax–Function–Aesthetics* reward mechanism, enabling the model to not only generate correct and functional code but also demonstrate a refined sense of **visual aesthetics**.
+On **ArtifactsBench**, [Ling-1T](https://zenmux.ai/inclusionai/ling-1t?utm_source=hf_inclusionAI) ranks **first among open-source models**, and the benchmark visualizations in this card were, in fact, *generated by Ling-1T itself*.
+
+
+### Emergent Intelligence at Trillion-Scale
+
+Scaling to the trillion-parameter level has revealed strong **emergent reasoning and transfer capabilities**.
+For example, in the **BFCL V3** tool-use benchmark, Ling-1T achieves **≈ 70% tool-call accuracy** with only light instruction tuning—despite having seen no large-scale trajectory data during training.
+[Ling-1T](https://zenmux.ai/inclusionai/ling-1t?utm_source=hf_inclusionAI) can:
+
+* Interpret complex natural-language instructions
+* Transform abstract logic into functional visual components
+* Generate cross-platform compatible front-end code
+* Create stylistically controlled marketing copy and multi-lingual text
+
+These capabilities form the foundation for **general, collaborative human–AI intelligence**, which we aim to advance together with the open-source community through Ling-1T’s release.
+
+
+### Pre-Training at Trillion Scale
+
+The Ling 2.0 architecture was designed from the ground up for trillion-scale efficiency, guided by the **Ling Scaling Law** ([arXiv:2507.17702](https://arxiv.org/abs/2507.17702)).
+This ensures architectural and hyperparameter scalability even under **1e25–1e26 FLOPs** of compute.
+
+Key architectural innovations include:
+
+* **1T total / 50B active parameters** with a **1/32 MoE activation ratio**
+* **MTP layers** for enhanced compositional reasoning
+* **Aux-loss-free**, **sigmoid-scoring expert routing** with **zero-mean updates**
+* **QK Normalization** for fully stable convergence
+
+<p align="center">
+    <img src="https://mdn.alipayobjects.com/huamei_bcz3yt/afts/img/naA9TJe7ttIAAAAAVRAAAAgADkV7AQFr/original"/>
+<p>
+
+Ling-1T is the **largest FP8-trained foundation model** known to date.
+FP8 mixed-precision training yields **15%+ end-to-end speedup**, improved memory efficiency, and maintains **≤ 0.1% loss deviation** from BF16 across **1T tokens**.
+A fine-grained, **heterogeneous 1F1B interleaved pipeline** further boosts utilization by 40 %+.
+System-level optimizations—fused kernels, communication scheduling, recomputation, checkpointing, simulation, and telemetry—ensure stable trillion-scale training.
+
+<p align="center">
+    <img src="https://mdn.alipayobjects.com/huamei_bcz3yt/afts/img/y5UVSKACgLEAAAAAVcAAAAgADkV7AQFr/original"/>
+<p>
+
+Pre-training used over **20T high-quality tokens**, with **> 40% reasoning-dense data** in later stages.
+Mid-training introduced **curated chain-of-thought corpora** for “**reasoning pre-activation**”, improving downstream reasoning stability.
+A custom **WSM (Warmup–Stable–Merge)** LR scheduler（[arXiv:2507.17634](https://arxiv.org/abs/2507.17634)） with mid-train checkpoint merging simulates LR decay and boosts generalization.
+
+
+### Post-Training and Evo-CoT Optimization
+
+Built upon mid-training reasoning activation, post-training adopts **Evo-CoT (Evolutionary Chain-of-Thought)** for progressive reasoning enhancement under controllable cost.
+This approach continually expands the **Pareto frontier** of reasoning accuracy vs. efficiency—ideal for reflexive non-thinking models.
+
+For reinforcement learning, we introduce **LPO (Linguistics-Unit Policy Optimization)** —a novel sentence-level policy optimization method.
+Unlike GRPO (token-level) or GSPO (sequence-level) algorithms, LPO treats *sentences* as the natural semantic action units, enabling precise alignment between rewards and reasoning behavior.
+Empirically, LPO offers superior **training stability** and **generalization** across reasoning tasks.
+
+<p align="center">
+    <img src="https://mdn.alipayobjects.com/huamei_bcz3yt/afts/img/kbEWT4BGEQQAAAAAWwAAAAgADkV7AQFr/original"/>
+<p>
+<p align="center">
+    <img src="https://mdn.alipayobjects.com/huamei_bcz3yt/afts/img/aF5LRqK5LMcAAAAAZHAAAAgADkV7AQFr/original"/>
+<p>
+
+## Evaluation
+
+Ling-1T has been extensively evaluated across **knowledge**, **code**, **math**, **reasoning**, **agent**, and **alignment** benchmarks.
+It currently stands as the **best open-source flagship non-thinking model**, rivaling closed-source APIs in complex reasoning while maintaining exceptional efficiency and interpretability.
+
+<p align="center">
+    <img src="https://mdn.alipayobjects.com/huamei_bcz3yt/afts/img/KrwiQZEDHV0AAAAAWkAAAAgADkV7AQFr/original"/>
+<p>
+
+
+## Model Downloads
+
+You can download Ling-1T from the following table. If you are located in mainland China, we also provide the model on ModelScope.cn to speed up the download process.
+
+<center>
+
+| **Model** | **Context Length** |                                                                 **Download**                                                                  |
+| :-------: | :----------------: | :-------------------------------------------------------------------------------------------------------------------------------------------: |
+|  Ling-1T  | 32K -> 128K (YaRN) | [🤗 HuggingFace](https://huggingface.co/inclusionAI/Ling-1T) &nbsp;&nbsp; [🤖 ModelScope](https://www.modelscope.cn/models/inclusionAI/Ling-1T) |
+
+</center>
+
+Note: If you are interested in previous version, please visit the past model collections in [Huggingface](https://huggingface.co/inclusionAI) or [ModelScope](https://modelscope.cn/organization/inclusionAI).
+
+
+## Quickstart
+
+### 🚀 Try Online
+
+You can experience Ling-1T online at: [ZenMux](https://zenmux.ai/inclusionai/ling-1t?utm_source=hf_inclusionAI)
+
+### 🔌 API Usage
+
+You can also use Ling-1T through API calls:
+
+```python   
+from openai import OpenAI
+
+# 1. Initialize the OpenAI client
+client = OpenAI(
+    # 2. Point the base URL to the ZenMux endpoint
+    base_url="https://zenmux.ai/api/v1",
+    # 3. Replace with the API Key from your ZenMux user console
+    api_key="<your ZENMUX_API_KEY>",
+)
+
+# 4. Make a request
+completion = client.chat.completions.create(
+    # 5. Specify the model to use in the format "provider/model-name"
+    model="inclusionai/ling-1t",
+    messages=[
+        {
+            "role": "user",
+            "content": "What is the meaning of life?"
+        }
+    ]
+)
+
+print(completion.choices[0].message.content)
+```
+
+## Deployment
+
+### SGLang
+
+#### Environment Preparation
+
+We will later submit our model to the SGLang official release. Now we can prepare the environment by following these steps:
+```shell
+pip3 install -U sglang sgl-kernel
+```
+
+#### Run Inference
+
+Both BF16 and FP8 models are supported by SGLang now. It depends on the dtype of the model in ${MODEL_PATH}.
+
+Here is the example to run Ling-1T with multiple GPU nodes, where the master node IP is ${MASTER_IP} and server port is ${PORT}:
+
+- Start server:
+```bash
+# Node 0:
+python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 0 
+
+# Node 1:
+python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 1 
+
+# Node 2:
+python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 2 
+
+# Node 3:
+python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 3
+
+# This is only an example. Please adjust arguments according to your actual environment.
+```
+
+- Client:
+
+```shell
+curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
+```
+
+More usage can be found [here](https://docs.sglang.ai/basic_usage/send_request.html)
+
+### vLLM
+
+#### Environment Preparation
+
+```bash
+pip install vllm==0.11.0
+```
+
+#### Run Inference:
+
+Here is the example to deploy the model with multiple GPU nodes, where the master node IP is ${MASTER_IP}, server port is ${PORT} and the path of model is ${MODEL_PATH}:
+
+```bash
+# step 1. start ray on all nodes
+
+# step 2. start vllm server only on node 0:
+vllm serve $MODEL_PATH --port $PORT --served-model-name my_model --trust-remote-code --tensor-parallel-size 8 --pipeline-parallel-size 4 --gpu-memory-utilization 0.85
+
+# This is only an example, please adjust arguments according to your actual environment.
+```
+
+To handle long context in vLLM using YaRN, we need to follow these two steps:
+1. Add a `rope_scaling` field to the model's `config.json` file, for example:
+```json
+{
+  ...,
+  "rope_scaling": {
+    "factor": 4.0,
+    "original_max_position_embeddings": 32768,
+    "type": "yarn"
+  }
+}
+```
+2. Use an additional parameter `--max-model-len` to specify the desired maximum context length when starting the vLLM service.
+
+For detailed guidance, please refer to the vLLM [`instructions`](https://docs.vllm.ai/en/latest/).
+
+
+
+## Limitations & Future Plans
+
+While **[Ling-1T](https://zenmux.ai/inclusionai/ling-1t?utm_source=hf_inclusionAI)** has made strong progress in efficient reasoning, cross-domain generalization, and training efficiency, several limitations remain:
+
+* **GQA-based attention**: stable for long-context reasoning but relatively costly. Future versions will adopt **hybrid attention** to improve efficiency.
+* **Limited agentic ability**: current model has room to grow in multi-turn interaction, long-term memory, and tool use.
+* **Instruction and identity issues**: occasional deviations or role confusion may occur; future updates will enhance **alignment and consistency**.
+
+The future versions of Ling-1T will continue to evolve in architecture, reasoning, and alignment, advancing the series toward more general intelligence.
+
+
+## License
+
+This code repository is licensed under [the MIT License](https://github.com/inclusionAI/Ling-V2/blob/main/LICENSE).
+
+## FAQ
+Recommended temperature? **0.7**  
+Recommended top_p? **0.95**

chat_template.jinja

@@ -0,0 +1,86 @@
+{% set thinking_option = 'off' %}
+{{- '<role>SYSTEM</role>' }}
+{%- if messages[0].role == 'system' %}
+    {{- messages[0].content + '\n' }}
+{%- endif %}
+{%- if tools %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>\n" }}
+{%- endif %}
+{{- 'detailed thinking ' + thinking_option + '<|role_end|>' }}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if message.role == "user" %}
+        {{- '<role>HUMAN</role>' + message.content + '<|role_end|>' }}
+    {%- elif message.role == "system" and not loop.first %}
+        {{- '<role>SYSTEM</role>' + message.content + '<|role_end|>' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = ((content.split('</think>')|first).rstrip('\n').split('<think>')|last).lstrip('\n') %}
+                {%- set content = (content.split('</think>')|last).lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {%- if reasoning_content %}
+                {{- '<role>ASSISTANT</role>' + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+            {%- else %}
+                {{- '<role>ASSISTANT</role>' + content }}
+            {%- endif %}
+        {%- else %}
+            {{- '<role>ASSISTANT</role>' + content }}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|role_end|>' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<role>OBSERVATION</role>' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|role_end|>' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<role>ASSISTANT</role>' }}
+{%- endif %}

config.json

@@ -0,0 +1,57 @@
+{
+  "architectures": [
+    "BailingMoeV2ForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_bailing_moe_v2.BailingMoeV2Config",
+    "AutoModel": "modeling_bailing_moe_v2.BailingMoeV2Model",
+    "AutoModelForCausalLM": "modeling_bailing_moe_v2.BailingMoeV2ForCausalLM"
+  },
+  "torch_dtype": "bfloat16",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 156892,
+  "first_k_dense_replace": 4,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 8192,
+  "initializer_range": 0.02,
+  "intermediate_size": 18432,
+  "max_position_embeddings": 32768,
+  "max_window_layers": 20,
+  "moe_intermediate_size": 2048,
+  "moe_router_enable_expert_bias": true,
+  "mtp_loss_scaling_factor": 0,
+  "n_group": 8,
+  "norm_head": false,
+  "norm_softmax": false,
+  "norm_topk_prob": true,
+  "num_attention_heads": 64,
+  "num_experts": 256,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 80,
+  "num_key_value_heads": 8,
+  "num_nextn_predict_layers": 0,
+  "num_shared_experts": 1,
+  "output_dropout": 0.0,
+  "output_router_logits": false,
+  "pad_token_id": 157150,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 600000,
+  "rotary_dim": 64,
+  "routed_scaling_factor": 2.5,
+  "router_dtype": "fp32",
+  "score_function": "sigmoid",
+  "tie_word_embeddings": false,
+  "topk_group": 4,
+  "transformers_version": "4.57.1",
+  "unsloth_fixed": true,
+  "use_bias": false,
+  "use_cache": true,
+  "use_qk_norm": true,
+  "use_qkv_bias": false,
+  "use_rmsnorm": true,
+  "using_split_qkv_in_self_attention": false,
+  "vocab_size": 157184
+}

configuration_bailing_moe_v2.py

@@ -0,0 +1,84 @@
+"""Bailing MoE V2 model configuration"""
+
+from transformers.configuration_utils import PretrainedConfig
+
+
+class BailingMoeV2Config(PretrainedConfig):
+
+    def __init__(
+        self,
+        vocab_size=157184,
+        hidden_size=2048,
+        intermediate_size=5120,
+        num_hidden_layers=20,
+        num_attention_heads=16,
+        num_key_value_heads=4,
+        hidden_act="silu",
+        use_qkv_bias=False,  # bailing only
+        use_bias=False,  # bailing only
+        rms_norm_eps=1e-06,
+        tie_word_embeddings=False,  # PretrainedConfig key, here change default value.
+        embedding_dropout=0.0,
+        attention_dropout=0.0,
+        output_dropout=0.0,
+        initializer_range=0.02,
+        max_position_embeddings=32768,
+        rope_theta=600000.0,
+        use_cache=True,
+        max_window_layers=20,
+        rope_scaling=None,
+        pad_token_id=156892,
+        eos_token_id=156892,
+        num_experts=256,
+        num_shared_experts=1,
+        num_experts_per_tok=8,
+        n_group=8,
+        topk_group=4,
+        moe_intermediate_size=512,
+        first_k_dense_replace=1,
+        head_dim=128,
+        output_router_logits=False,
+        use_qk_norm=True,
+        num_nextn_predict_layers=0,
+        mtp_loss_scaling_factor=0,
+        moe_router_enable_expert_bias=True,
+        routed_scaling_factor=1.0,
+        **kwargs,
+    ):
+        self.num_hidden_layers = num_hidden_layers
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_attention_heads = num_attention_heads
+        self.num_key_value_heads = num_key_value_heads
+        self.hidden_act = hidden_act
+        self.use_qkv_bias = use_qkv_bias
+        self.use_bias = use_bias
+        self.rms_norm_eps = rms_norm_eps
+        self.embedding_dropout = embedding_dropout
+        self.attention_dropout = attention_dropout
+        self.output_dropout = output_dropout
+        self.num_nextn_predict_layers = num_nextn_predict_layers
+        self.mtp_loss_scaling_factor = mtp_loss_scaling_factor
+        self.initializer_range = initializer_range
+        self.max_position_embeddings = max_position_embeddings
+        self.rope_theta = rope_theta
+        self.use_cache = use_cache
+        self.max_window_layers = max_window_layers
+        self.head_dim = head_dim or self.hidden_size // self.num_attention_heads
+        self.rope_scaling = rope_scaling
+        self.use_qk_norm = use_qk_norm
+        self.moe_router_enable_expert_bias = moe_router_enable_expert_bias
+        self.routed_scaling_factor = routed_scaling_factor
+
+        # MoE configs
+        self.num_experts = num_experts
+        self.num_shared_experts = num_shared_experts
+        self.num_experts_per_tok = num_experts_per_tok
+        self.n_group = n_group
+        self.topk_group = topk_group
+        self.moe_intermediate_size = moe_intermediate_size
+        self.first_k_dense_replace = first_k_dense_replace
+        self.output_router_logits = output_router_logits
+
+        super().__init__(pad_token_id=pad_token_id, eos_token_id=eos_token_id, tie_word_embeddings=tie_word_embeddings, **kwargs)

generation_config.json

@@ -0,0 +1,9 @@
+{
+    "bos_token_id": 156891,
+    "eos_token_id": [
+        156892,
+        156895
+    ],
+    "pad_token_id": 156892,
+    "transformers_version": "4.52.3"
+}

model-00001-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02b5cbbf76f7b21e54a936db254aef54aa4196d8a729c93ce6e26e7467c9448c
+size 2575302800

model-00002-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8bd28b5ae46d9bfd9a606fb9a3d78145a82ccc683cba6046d0778c5cff62774b
+size 17179933360

model-00003-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34fe8fa1233a826f93c2017ed28346c917d208acba742f9e499bdf862e29a15a
+size 10708107872

model-00004-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4228da5f6d58a79560d41a80f697562ff44f7bbb9803a5890783fececb0c601d
+size 3288489872

model-00005-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49a3a53e41ac87a436a1786c24d0323287974562c92548307770dcb2b3b422dd
+size 17179933360

model-00006-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8ec47016f9a3d68b949eb59194c1721855edaa58e5c06f6f1de110a1b9f5779
+size 8657075832

model-00007-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:01cf04ff48fe7d7d9a2770a8a95ac9cdd0e64b58b1197035a490f7f766c64602
+size 17179933360

model-00008-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:333488f4b27ff934a3fa3164da9325320c0944fad308da2e478b30e4a729a14d
+size 8589966712

model-00009-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:008d73992a834cefbfc1f2f56ecdb32d1738f68d28cd2df4157f3cae5a0107d4
+size 17179933360

model-00010-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f0aecdfbf532e98b935915b0841d01be6960a6e1716ff58833acd5704f4ce6a
+size 8657075832

model-00011-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a488ad4f161015436e9f4ff84a9777e0778043aa01aa5b46bfea8f5fd9182ce3
+size 17179933360

model-00012-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:383093458cbad7d3fc8ab944a5d9e47ef9e01b514e1558bdcafa46fe6cf5cc7c
+size 8589966712

model-00013-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2069456d5f45c0ab79c493504d101b908a3cabb03ff16434254924835312e3eb
+size 17179933360

model-00014-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6432062b2480faf01fd834cc3a56d23730f63a3ceb2edb2cc38473282e8b8a69
+size 10490163984

model-00015-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c585d0611f633e6005a1aaff570c2063316a02995a85c5de37b34c73cc23189d
+size 17179933872

model-00016-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2775dcc6573a9473beb946bd0166c4a01f2f6f9b8d5ccbd3facffd778e6f9250
+size 8657076088

model-00017-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3619ed11511840a7bb5b123505d2cff512c72363a23aac618258a13d6c7d1ef0
+size 17179933872

model-00018-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d59267abd6626bf0e90a35b9fa30cf6c7d887a6fad811c92029fd4c8ce27083a
+size 8589966968

model-00019-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:21adbbd374b63a8bc5e50f9466ce0aad7d5246f84c3daaeee43d70b65bbee154
+size 17179933872

model-00020-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad841c7b90c64ec1050762ae562627b7ec9f9ab132a66d8e88e6338439b24d37
+size 8657076088

model-00021-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:380b6583db0da7e30a87a5b8e096d5773085a4d53ede7498e23735c934d03490
+size 17179933872

model-00022-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d50cc291578e241e0af278fbdf34b196baaa1620b99b027c4536ce5309af803a
+size 8589966968

model-00023-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f2c73f58ef51962d88c22536b807ba0fd3c01544209e2a686dc0c65ad132c2e
+size 17179933872

model-00024-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a08d3eeb95f6289dd9af1e441cf4c05891f9c764ab7e990c6e76e2b4af86c22d
+size 10490164272

model-00025-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6878d24020df6bf0453a99fe3011a210a5f55e4acced37d3634e3e057ea99dbc
+size 17179933872

model-00026-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77282b6a5d29a1c70401004d0e05e614379a4cf0d95d804ef957207bf6899222
+size 8657076088

model-00027-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:281173419752769f7f75df827e1ae6fd2f0eaf8b45264e7c671221a7b64d7428
+size 17179933872

model-00028-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1688f9e2ec553ad717fd0d66aee51a5940e18c4ab92c5b7854a43490e0e24fb9
+size 8589966968

model-00029-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:db40bd9419270afc67edb8d0a8dc69d9d1099f892b9ae11fdc1242d0b9279a3e
+size 17179933872

model-00030-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c403ade0ebdd0054d297541f878bfedad6bdfc9ed22978eeab61d8efdb415482
+size 8657076088

model-00031-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e883b038c46d2aeba5684db61d230b6350e89b1ac0ee143ed73d0096a49b70dd
+size 17179933872

model-00032-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f885a5608a7158d6f0a55d8e20ace0f22cecca25f54a66069c66e08df5159061
+size 8589966968

model-00033-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ecb9d25417c4b5c8b38dae83b1fb545e691e1b73d906fa04ca0ecad78c22aad
+size 17179933872

model-00034-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3778e8558dae08ae7d15f1a56a9336fc54080365ea614ddeefddc1f73d10904e
+size 10490164272

model-00035-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83952cf51e885baf669717b6715df137b346a6a30d0ca10a8cb8cb8e7601cefa
+size 17179933872

model-00036-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9634b8b63c9aa27b4d26de509c35a3573faad4f480881cd06a338e78c7319371
+size 8657076088

model-00037-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5006fc1db0febe78c7aa5806fa2216366e8eeaa1ce35490eddb0e4b19075322
+size 17179933872

model-00038-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c749514f6a116cd9e92fde5806fd1d8f9ddf9cc8109d5ee4e661048cbf30e7ac
+size 8589966968

model-00039-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d7795ccc126467aaaf1b541ef57b67a63815095468beed6b677cd1babe650002
+size 17179933872

model-00040-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65a272b0c3386985097812a3d6cd32eb3e338725d1870bc9ed36915d241b24c5
+size 8657076088

model-00041-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2495349816f2a677d8ef73711ba8f9094917ec2ce7b27c01c7fce6bdd9d1cfcb
+size 17179933872

model-00042-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:66f7ac37d2d6bea6df36ab4596ee402242d3037abea2c15fe1377e92281b89f6
+size 8589966968

model-00043-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eaf78374089e06fec5abb6838efb0e7fc1a6c6c38bc840a123a5aa1b1cf82197
+size 17179933872

model-00044-of-00155.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d36c97a6e7a86c5796656a88abd1171a403937c11490c837e4eb382c6559be67
+size 10490164272