Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,284 +1,36 @@
 ---
 license: gemma
 pipeline_tag: text-generation
+base_model: scb10x/typhoon2.1-gemma3-12b
+library_name: mlx
+tags:
+- mlx
 ---
 
-**Typhoon2.1-Gemma3-12B**: Thai Large Language Model (Instruct)
+# scb10x/typhoon2.1-gemma3-12b-mlx-4bit
 
-**Typhoon2.1-Gemma3-12B** is a instruct Thai 🇹🇭 large language model with 12 billion parameters, a 128K context length, and function-calling capabilities. It is based on Gemma3 12B.
+This model [scb10x/typhoon2.1-gemma3-12b-mlx-4bit](https://huggingface.co/scb10x/typhoon2.1-gemma3-12b-mlx-4bit) was
+converted to MLX format from [scb10x/typhoon2.1-gemma3-12b](https://huggingface.co/scb10x/typhoon2.1-gemma3-12b)
+using mlx-lm version **0.25.2**.
 
-Remark: This is text only model. We removed vision encoder for this version due to complexity. Stay-tune for version with vision encoder soon.
-
-## **Performance**
-
-![12b model performance](https://storage.googleapis.com/typhoon-public/assets/typhoon-21/performance12b_table.png)
-
-## **Model Description**
-
-- **Model type**: A 12B instruct decoder-only model based on Gemma3 architecture.
-- **Requirement**: transformers 4.50.0 or newer.
-- **Primary Language(s)**: Thai 🇹🇭 and English 🇬🇧
-- **Context Length**: 128K
-- **License**: [Gemma License](https://github.com/google-deepmind/gemma/blob/main/LICENSE)
-
-
-## Usage Example
-
-This code snippet shows how to use the Typhoon2.1-Gemma3-12B model for Thai or English text generation using the transformers library. It includes setting up the model and tokenizer, formatting chat messages in a system-user style, and generating a response.
-
-```python
-from transformers import AutoTokenizer, AutoModelForCausalLM
-import torch
-
-model_id = "scb10x/typhoon2.1-gemma3-12b"
-
-tokenizer = AutoTokenizer.from_pretrained(model_id)
-model = AutoModelForCausalLM.from_pretrained(
-    model_id,
-    torch_dtype=torch.bfloat16,
-    device_map="auto",
-)
-
-messages = [
-    {"role": "system", "content": "You are a male AI assistant named Typhoon created by SCB 10X to be helpful, harmless, and honest. Typhoon is happy to help with analysis, question answering, math, coding, creative writing, teaching, role-play, general discussion, and all sorts of other tasks. Typhoon responds directly to all human messages without unnecessary affirmations or filler phrases like “Certainly!”, “Of course!”, “Absolutely!”, “Great!”, “Sure!”, etc. Specifically, Typhoon avoids starting responses with the word “Certainly” in any way. Typhoon follows this information in all languages, and always responds to the user in the language they use or request. Typhoon is now being connected with a human. Write in fluid, conversational prose, Show genuine interest in understanding requests, Express appropriate emotions and empathy. Also showing information in term that is easy to understand and visualized."},
-    {"role": "user", "content": "ขอสูตรไก่ย่าง"},
-]
-
-input_ids = tokenizer.apply_chat_template(
-    messages,
-    add_generation_prompt=True,
-    return_tensors="pt",
-    enable_thinking=False # Switches between thinking and non-thinking modes. Default is False.
-).to(model.device)
-
-outputs = model.generate(
-    input_ids,
-    max_new_tokens=512,
-    do_sample=True,
-    temperature=0.6,
-    top_p=0.95,
-)
-response = outputs[0][input_ids.shape[-1]:]
-print(tokenizer.decode(response, skip_special_tokens=True))
-```
-
-## Deploy as Server
-
-This section shows how to run Typhoon2.1 as an OpenAI-compatible API server using vllm.
+## Use with mlx
 
 ```bash
-pip install vllm
-vllm serve scb10x/typhoon2.1-gemma3-12b --max-model-len 16000 --dtype bfloat16 --tool-call-parser pythonic --enable-auto-tool-choice 
-# adjust --max-model-len based on your avaliable memory
-# you can use --quantization bitsandbytes to reduce the memory use while trade-off inference speed
-```
-
-## Using Tools
-
-You can provide tools to the vLLM-powered OpenAI-compatible API for functionality.
-
+pip install mlx-lm
 ```
-from openai import OpenAI
-import json
-
-client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
-
-def get_weather(location: str, unit: str):
-    return f"Getting the weather for {location} in {unit}..."
-tool_functions = {"get_weather": get_weather}
-
-tools = [{
-    "type": "function",
-    "function": {
-        "name": "get_weather",
-        "description": "Get the current weather in a given location",
-        "parameters": {
-            "type": "object",
-            "properties": {
-                "location": {"type": "string", "description": "City and state, e.g., 'San Francisco, CA'"},
-                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
-            },
-            "required": ["location", "unit"]
-        }
-    }
-}]
-
-response = client.chat.completions.create(
-    model=client.models.list().data[0].id,
-    messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],
-    tools=tools,
-    tool_choice="auto"
-)
-
-tool_call = response.choices[0].message.tool_calls[0].function
-print(f"Function called: {tool_call.name}")
-print(f"Arguments: {tool_call.arguments}")
-print(f"Result: {get_weather(**json.loads(tool_call.arguments))}")
-```
-
-
-## Switching Between Thinking and Non-Thinking Mode
-
-Typhoon supports two modes:
-Non-thinking mode (default): Fast response generation without extra reasoning steps.
-Thinking mode: The model first reasons internally, then provides a clearer and potentially more accurate final answer.
-You can enable thinking mode by:
-Setting enable_thinking=True in apply_chat_template.
-Using a special system prompt that instructs the model to reason inside <think>...</think> tags.
-
-You can turn on thinking mode by either
-- add enable_thinking=True to apply_chat_template
 
 ```python
-input_ids = tokenizer.apply_chat_template(
-    messages,
-    add_generation_prompt=True,
-    return_tensors="pt",
-    enable_thinking=True # Switches between thinking and non-thinking modes. Default is False.
-).to(model.device)
-```
-
-- manually by supply thinking mode system prompt
-
-```
-You are a helpful assistant. First, think through the reasoning internally, then present the reasoning within <think>...</think>. After thinking, clearly state a response that addresses the user's request and aligns with their preferences, not just providing a direct answer.
-```
-
-- in vllm powered openai compatible client you can add chat_template_kwargs to the post payload
-```json
-{
-  "model": "scb10x/typhoon2.1-gemma3-12b",
-  "messages": [
-    {"role": "user", "content": "Give me a short introduction to large language models."}
-  ],
-  "chat_template_kwargs": {"enable_thinking": true}
-}
-```
-
-## Budget forcing
-
-This section introduces budget forcing, an advanced technique to let the model spend more time and tokens reasoning before producing a final answer—great for improving performance on complex questions.
-
-```
-from vllm import LLM, SamplingParams
-from transformers import AutoTokenizer
-class BudgetForcingHandler:
-    
-    def __init__(self, model_name: str, max_think_token: int, max_ignore=5, temperature=0.6, seed=32):
-        self.temperature = temperature
-        self.seed = seed
-        self.max_think_token = max_think_token
-        self.max_ignore = max_ignore
-        self.model = LLM(model_name, dtype='bfloat16', enforce_eager=True)
-        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
-        self.alternative_str = '\nAlternatively'
-        self.system = """You are a reasoning assistant. First, think through the reasoning internally, then present the reasoning within <think>...</think>. After thinking, clearly state the final answer."""
-    
-    def __call__(self, prompts: List[str]):
-        count_prompt = len(prompts)
-        prompts = [self.tokenizer.apply_chat_template([{'role': 'system', 'content': self.system}, {'role': 'user', 'content': f'Please solve this math question, and put your final answer within \\boxed{{}}.\n{p}'}], add_generation_prompt=True, tokenize=False) for p in prompts]
-        sampling_params = SamplingParams(
-            max_tokens=self.max_think_token,
-            seed=self.seed,
-            stop=["</think>"],
-            skip_special_tokens=False,
-            temperature=self.temperature,
-        )
-        o = self.model.generate(
-            prompts,
-            sampling_params=sampling_params
-        )
-        
-        outputs = [output.outputs[0].text for output in o]
-        token_count = [len(output.outputs[0].token_ids) for output in o]
-        for i in range(len(prompts)):
-            prompts[i] = prompts[i] + outputs[i]
-        
-        for _ in range(self.max_ignore): # Num of times to skip stop token
-            inference_loop_prompts = []
-            inference_idx = []
-            max_inference_token = 0
-            print('current token count: ', token_count)
-            for i in range(len(prompts)):
-                left_budget = self.max_think_token - token_count[i]
-                if left_budget > 0:
-                    prompts[i] = prompts[i] + self.alternative_str
-                    inference_loop_prompts.append(prompts[i])
-                    inference_idx.append(i)
-                    if left_budget > max_inference_token:
-                        max_inference_token = left_budget
-            
-            outputs = ['' for _ in range(len(prompts))]
-            if max_inference_token == 0 or len(inference_loop_prompts) == 0:
-                break
-            sampling_params = SamplingParams(
-                max_tokens=max_inference_token,
-                min_tokens=1,
-                seed=self.seed,
-                stop=["</think>"],
-                skip_special_tokens=False,
-                temperature=self.temperature,
-            )
-            o = self.model.generate(
-                inference_loop_prompts,
-                sampling_params=sampling_params
-            )
-            assert len(inference_idx) == len(inference_loop_prompts)
-            assert len(inference_idx) == len(o)
-            for i, output in zip(inference_idx, o):
-                outputs[i] = output.outputs[0].text
-            
-            for i, idx in enumerate(inference_idx):
-                token_count[idx] = token_count[idx] + len(o[i].outputs[0].token_ids)
-            
-            for i in range(len(prompts)):
-                prompts[i] = prompts[i] + outputs[i]
-        print('generating answer...')
-        prompts = [p + '\nTime\'s up. End of thinking process. Will answer immediately.\n</think>' for i, p in enumerate(prompts)]
-        sampling_params = SamplingParams(
-            max_tokens=2048,
-            min_tokens=0,
-            seed=self.seed,
-            skip_special_tokens=False,
-            temperature=self.temperature,
-        )
-        o = self.model.generate(
-            prompts,
-            sampling_params=sampling_params,
-        )
-        for i in range(len(prompts)):
-            prompts[i] = prompts[i] + o[i].outputs[0].text
-        assert len(prompts) == count_prompt
-        return prompts
-
-handler = BudgetForcingHandler("scb10x/typhoon2.1-gemma3-12b", max_think_token=2048)
-handler(["How many r in raspberry?"])
-```
-
-## **Intended Uses & Limitations**
-
-This model is an instructional model. However, it’s still undergoing development. It incorporates some level of guardrails, but it still may produce answers that are inaccurate, biased, or otherwise objectionable in response to user prompts. We recommend that developers assess these risks in the context of their use case.
-
-## **Follow us**
-
-**https://twitter.com/opentyphoon**
-
-## **Support**
+from mlx_lm import load, generate
 
-**https://discord.gg/us5gAYmrxw**
+model, tokenizer = load("scb10x/typhoon2.1-gemma3-12b-mlx-4bit")
 
+prompt = "hello"
 
-## **Citation**
+if tokenizer.chat_template is not None:
+    messages = [{"role": "user", "content": prompt}]
+    prompt = tokenizer.apply_chat_template(
+        messages, add_generation_prompt=True
+    )
 
-- If you find Typhoon2 useful for your work, please cite it using:
+response = generate(model, tokenizer, prompt=prompt, verbose=True)
 ```
-@misc{typhoon2,
-      title={Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Models}, 
-      author={Kunat Pipatanakul and Potsawee Manakul and Natapong Nitarach and Warit Sirichotedumrong and Surapon Nonesung and Teetouch Jaknamon and Parinthapat Pengpun and Pittawat Taveekitworachai and Adisai Na-Thalang and Sittipong Sripaisarnmongkol and Krisanapong Jirayoot and Kasima Tharnpipitchai},
-      year={2024},
-      eprint={2412.13702},
-      archivePrefix={arXiv},
-      primaryClass={cs.CL},
-      url={https://arxiv.org/abs/2412.13702}, 
-}
-```

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "<image_soft_token>": 262144
+}

chat_template.jinja

@@ -0,0 +1,146 @@
+{#- Begin-of-sequence token to start the model prompt -#}
+{{ bos_token }}
+{#- Extracts the system message. Gemma does not support system messages so it will be prepended to first user message. -#}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "You are a helpful assistant named Typhoon created by SCB 10X to be helpful, harmless, and honest." -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- if enable_thinking is defined and enable_thinking is true %}
+    {%- set first_user_prefix = first_user_prefix + " First, think through the reasoning internally, then present the reasoning within <think>...</think>. After thinking, clearly state a response that addresses the user's request and aligns with their preferences, not just providing a direct answer." -%}
+{%- endif %}
+{%- set first_user_prefix = first_user_prefix + '\n\n' -%}
+{#- Set tools to none if not defined for this ChatCompletion request (helps avoid errors later) -#}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+
+{#- If given only system message -#}
+{%- if loop_messages|length == 0 -%}
+    {{ '<start_of_turn>user\n' -}}
+    {{ first_user_prefix }}
+    {#- Append system message with tool information if using tools in message request. -#}
+    {%- if tools is not none -%}
+        {{- "Tools (functions) are available. If you decide to invoke one or more of the tools, you must respond with a python list of the function calls.\n" -}}
+        {{- "Example Format: [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)] \n" -}}
+        {{- "Do not use variables. DO NOT USE MARKDOWN SYNTAX. You SHOULD NOT include any other text in the response if you call a function. If none of the functions can be used, point it out. If you lack the parameters required by the function, also point it out.\n" -}}
+        {{- "Here is a list of functions in JSON format that you can invoke.\n" -}}
+        {{- tools | tojson(indent=4) -}}
+        {{- "\n\n" -}}
+    {%- endif -%}
+{%- endif %}
+
+{#- Main loop over all messages in the conversation history -#}
+{%- for message in loop_messages -%}
+    {#- Normalize roles for model prompt formatting -#}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- elif (message['role'] == 'tool') -%}
+        {%- set role = "user" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {#- Mark the start of a message block with the appropriate role -#}
+    {{ '<start_of_turn>' + role + '\n' -}}
+
+    {#- Insert system message content (if present) at the beginning of the first message. -#}
+    {%- if loop.first -%}
+        {{ first_user_prefix }}
+        {#- Append system message with tool information if using tools in message request. -#}
+        {%- if tools is not none -%}
+            {{- "Tools (functions) are available. If you decide to invoke one or more of the tools, you must respond with a python list of the function calls.\n" -}}
+            {{- "Example Format: [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)] \n" -}}
+            {{- "Do not use variables. DO NOT USE MARKDOWN SYNTAX. You SHOULD NOT include any other text in the response if you call a function. If none of the functions can be used, point it out. If you lack the parameters required by the function, also point it out.\n" -}}
+            {{- "Here is a list of functions in JSON format that you can invoke.\n" -}}
+            {{- tools | tojson(indent=4) -}}
+            {{- "\n\n" -}}
+        {%- endif -%}
+    {%- endif -%}
+
+    {#- Format model tool calls (turns where model indicates they want to call a tool) -#}
+    {%- if 'tool_calls' in message -%}
+        {#- Opening bracket for tool call list. -#}
+        {{- '[' -}}
+        {#- For each tool call -#}
+        {%- for tool_call in message.tool_calls -%}
+            {#- Function name & opening parenthesis. -#}
+            {%- if tool_call.function is defined -%}
+                {%- set tool_call = tool_call.function -%}
+            {%- endif -%}
+            {{- tool_call.name + '(' -}}
+
+            {#-- Handle arguments as list (positional) or dict (named) --#}
+            {#-- Named arguments (dict) --#}
+            {%- if tool_call.arguments is iterable and tool_call.arguments is mapping -%}
+                {%- set first = true -%}
+                {%- for key, val in tool_call.arguments.items() -%}
+                    {%- if not first %}, {% endif -%}
+                    {{ key }}={{ val | tojson }}
+                    {%- set first = false -%}
+                {%- endfor -%}
+            {#-- Positional arguments (list) --#}
+            {%- elif tool_call.arguments is iterable -%}
+                {{- tool_call.arguments | map('tojson') | join(', ') -}}
+            {#-- Fallback: single positional value --#}
+            {%- else -%}
+                {{- tool_call.arguments | tojson -}}
+            {#-- Closing parenthesis. --#}
+            {%- endif -%}
+                {{- ')' -}}
+            {#-- If more than one tool call, place comma and move to formatting next tool call --#}
+            {%- if not loop.last -%}, {% endif -%}
+        {%- endfor -%}
+        {#- Closing bracket for tool call list. -#}
+        {{- ']' -}}
+    {%- endif -%}
+    
+    {#- Tool response start tag (for messages from a tool) -#}
+    {%- if (message['role'] == 'tool') -%}
+        {{ '<tool_response>\n' -}}
+    {%- endif -%}
+
+    {#- Render the message content: handle plain string or multimodal content like image/text -#}
+    {%- if message['content'] is string -%}
+        {%- set content = message['content'] -%}
+        {%- if '</think>' in content -%}
+            {%- set content = content.split('</think>')[-1] -%}
+        {%- endif -%}
+        {{ content | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {%- set content = item['text'] -%}
+                {%- if '</think>' in content -%}
+                    {%- set content = content.split('</think>')[-1] -%}
+                {%- endif -%}
+                {{ content | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+
+    {#- Tool response end tag -#}
+    {%- if (message['role'] == 'tool') -%}
+        {{ '</tool_response>' -}}
+    {%- endif -%}
+
+    {#- Mark end of a single turn -#}
+    {{ '<end_of_turn>\n' }}
+{%- endfor -%}
+
+{#- If generation is to be triggered, add model prompt prefix -#}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model\n'}}
+    {%- if enable_thinking is defined and enable_thinking is true -%}
+        {{- '<think>' -}}
+    {%- endif %}
+{%- endif -%}

config.json

@@ -0,0 +1,45 @@
+{
+    "architectures": [
+        "Gemma3ForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_logit_softcapping": null,
+    "bos_token_id": 2,
+    "cache_implementation": "hybrid",
+    "eos_token_id": 1,
+    "final_logit_softcapping": null,
+    "head_dim": 256,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 3840,
+    "initializer_range": 0.02,
+    "intermediate_size": 15360,
+    "max_position_embeddings": 131072,
+    "model_type": "gemma3_text",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 48,
+    "num_key_value_heads": 8,
+    "pad_token_id": 0,
+    "quantization": {
+        "group_size": 64,
+        "bits": 4
+    },
+    "quantization_config": {
+        "group_size": 64,
+        "bits": 4
+    },
+    "query_pre_attn_scalar": 256,
+    "rms_norm_eps": 1e-06,
+    "rope_local_base_freq": 10000.0,
+    "rope_scaling": {
+        "factor": 8.0,
+        "rope_type": "linear"
+    },
+    "rope_theta": 1000000.0,
+    "sliding_window": 1024,
+    "sliding_window_pattern": 6,
+    "torch_dtype": "bfloat16",
+    "transformers_version": "4.50.0.dev0",
+    "use_cache": false,
+    "vocab_size": 262145
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "pad_token_id": 0,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "4.50.0.dev0"
+}

model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82e76df80a7af51acb31081635a3ad154267585b3bff306b5f854491f4156893
+size 5367250560

model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62e9dd9bb0ce30408a5cf16afe9d39e2c1baeada788756015818c8a2f396b260
+size 1818490740

special_tokens_map.json

@@ -0,0 +1,33 @@
+{
+  "boi_token": "<start_of_image>",
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eoi_token": "<end_of_image>",
+  "eos_token": {
+    "content": "<end_of_turn>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<image_soft_token>",
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a872e3bb510a751b26bd65f61aad05f948c9cf78fe4f787aebd197b393cc4081
+size 33384667

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
+size 4689074