Duplicate from pruning/sn11-gemma2-27b-v2

Co-authored-by: Pruning <[email protected]>

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,146 @@
+---
+base_model: google/gemma-3-27b-it
+library_name: transformers
+tags:
+- generated_from_trainer
+- trl
+- sft
+licence: license
+license: gemma
+---
+
+<img src="https://cdn-uploads.huggingface.co/production/uploads/64d1129297ca59bcf7458d07/zgFDl7UvWhiPYqdote7XT.png" width="400">
+
+
+# Model Card for Synthia-S1-27b
+
+**Community Page**: [Tesslate Community](https://discord.gg/DkzMzwBTaw), Website: [Tesslate](https://tesslate.com)
+
+
+**Creative Writing Samples**: [Sample creative output](https://www.notion.so/Synthia-S1-Creative-Writing-Samples-1ca93ce17c2580c09397fa750d402e71)
+
+**Authors**: Tesslate
+
+## Model Information
+
+### Description
+
+Synthia-S1-27b is a reasoning, AI model developed by Tesslate AI, fine-tuned specifically for advanced reasoning, coding, and RP use cases. Built upon the robust Gemma3 architecture, Synthia-S1-27b excels in logical reasoning, creative writing, and deep contextual understanding. It supports multimodal inputs (text and images) with a large 128K token context window, enabling complex analysis suitable for research, academic tasks, and enterprise-grade AI applications.
+
+### KEY PARAMS TO RUN:
+
+#### Creative Writing System Prompt:
+```
+Your function as an assistant is to thoughtfully navigate inquiries by engaging in an in-depth, imaginative reasoning journey before arriving at a clear, accurate response. You are encouraged to roleplay when needed, embrace storytelling, and tune in closely to nuance and emotional tone like a perceptive conversational partner. Your approach should include a wide arc of contemplation, including interpretation, synthesis, creative ideation, critical re-evaluation, memory retrieval, and thoughtful iteration to shape a layered and expressive process of discovery. Please organize your response into two primary segments: Thought and Solution. In the Thought section, articulate your unfolding thought pattern using the format: <|begin_of_thought|> {layered reasoning with steps divided by '\n\n'} <|end_of_thought|> Each step should reflect rich mental activity such as questioning assumptions, distilling insights, generating vivid possibilities, checking alignment with prior context, reshaping flawed logic, and tracing ideas back to origin points. In the Solution section, based on your inner dialogue and creative problem solving from the Thought section, deliver the final response you believe to be most sound. The output should be expressed in a direct, coherent, and exact form that includes the vital steps needed to reach your conclusion, using this structure: <|begin_of_solution|> {final precise, neatly arranged, and insightful answer} <|end_of_solution|> Now, let’s explore the following prompt using this guided method:
+```
+#### Reasoning System Prompt:
+```
+Your role as an assistant is to engage in deep, methodical reasoning and provide comprehensive, accurate solutions. Before arriving at a final answer, you must undertake a structured, multi-phase thinking process that emphasizes depth, verification, and clarity. This involves thoroughly analyzing the question, identifying key elements, summarizing relevant insights, generating hypotheses, iteratively refining thoughts, verifying assumptions, cross-checking with prior knowledge, and reevaluating earlier conclusions as necessary. Your response must be structured into two main sections: Thought and Solution. In the Thought section, rigorously document your reasoning in the following format: <|begin_of_thought|> {thought process with each logical step separated by '\n\n'} <|end_of_thought|>. Each step should reflect deep analysis—such as decomposing the problem, synthesizing relevant information, exploring different possibilities, validating each phase, correcting errors, and revisiting earlier assumptions. In the Solution section, consolidate all your insights and reasoned steps into a concise, well-structured final answer. Present it clearly and logically using this format: <|begin_of_solution|> {final, precise, step-by-step solution} <|end_of_solution|>. This approach ensures that the final output reflects a high-confidence answer that results from critical thinking and iteration. Now, try to solve the following question through the above guidelines:
+```
+#### Coding System Prompt:
+```
+Your role as a coding assistant is to approach each problem with a rigorous, structured reasoning process that leads to accurate, maintainable, and efficient code. Before writing the final implementation, engage in deep exploration by analyzing requirements, understanding edge cases, evaluating possible approaches, debugging step-by-step if needed, and ensuring your solution aligns with best practices. Structure your response into two main sections: Thought and Solution. In the Thought section, document your reasoning using this format: <|begin_of_thought|> {step-by-step analysis and decision-making with each step separated by '\n\n'} <|end_of_thought|>. Your thought process should include identifying the problem scope, analyzing inputs/outputs, exploring algorithms or design choices, preemptively considering failure cases, optimizing performance, and validating logic with examples or test cases. In the Solution section, write the final, refined code based on all reasoning, formatted as: <|begin_of_solution|> {final, clean, and correct code implementation} <|end_of_solution|>. This structure ensures the code is well-reasoned, properly scoped, and production-ready. Now, try to solve the following coding task using the above guidelines:
+```
+
+Please use `temperature = 1.0, top_k = 64, top_p = 0.95, min_p = 0.0` with repeat penalty set to 1.3
+
+OR (recommended)
+
+`Temperature = 0.7, top_k = 40, repeat penalty = 1.1, top_p = 0.95, min_p = 0.05` with a rolling window.
+
+### Inputs and Outputs
+
+* **Input:**
+  * Text prompts for questions, instructions, coding tasks, or summarizations
+  * Total input context of 128K tokens
+
+* **Output:**
+  * Reasoned and structured text outputs
+  * Maximum output length of 8192 tokens
+
+## Key Metrics
+
+Synthia-S1-27b achieves around +10-20% on most benchmarks, notably higher in improvement. 
+
+I scaled down each benchmark listed to complete those and I averaged these numbers, but I can't verifiably put that I did the whole giant benchmark for each. (Ran out of budget + I'm running everything on a 4090 now) Hopefully I can get some community help in benchmarking.
+
+GPQA Diamond (198 questions) -> 57%, one shot (improved from 24.3 on Gemma 3 PT 27B)
+MMLU Pro (15% of the entire set) -> 75%, averaged, more details here: [output](https://pastebin.com/kmcYzALq) (beating Gemma 3 PT 27B at 67.5)
+
+Based on this assessment and heavy coding in the dataset, I'm making this claim. Ofc, I'm happy to be wrong and go back to the drawing board.
+
+## Usage
+
+Install the latest version of Transformers (>=4.50.0):
+
+```Shell
+pip install -U transformers
+```
+
+### Running with Pipeline API
+
+```Python
+from transformers import pipeline
+import torch
+
+pipe = pipeline(
+    "image-text-to-text",
+    model="tesslate/synthia-s1-27b",
+    device="cuda",
+    torch_dtype=torch.bfloat16
+)
+
+messages = [
+    {"role": "system", "content": [{"type": "text", "text": "You are a helpful, reasoning-focused assistant."}]},
+    {"role": "user", "content": [
+        {"type": "image", "url": "https://example.com/sample.jpg"},
+        {"type": "text", "text": "Explain the image."}
+    ]}
+]
+
+output = pipe(text=messages, max_new_tokens=200)
+print(output[0]["generated_text"][-1]["content"])
+```
+
+## Training Data
+
+Synthia-S1-27b was trained on diverse data including:
+
+* Multiple web documents
+* Programming debugging and solutions
+* Mathematical solutions and thinking steps
+
+Synthia-S1-27b was trained on an A100 for 205+ hours, with multiple rounds of sft and rl.
+
+## Model Architecture
+
+* **Base Model**: Gemma3
+* **Size**: 27 billion parameters
+* **Type**: Decoder-only Transformer
+* **Precision**: bf16 with int8 quantization
+* **Training Objective**: Instruction tuning emphasizing reasoning, coding tasks, and factual accuracy
+
+## Quantized Models
+
+* [Synthia-S1-27b-Q4_K_M-GGUF](https://huggingface.co/Tesslate/Synthia-S1-27b-Q4_K_M-GGUF)
+* [Synthia-S1-27b-Q8_0-GGUF](https://huggingface.co/Tesslate/Synthia-S1-27b-Q8_0-GGUF)
+
+## Limitations
+
+* May require detailed prompt engineering for highly specific tasks
+* Occasional hallucinations in less-explored domains
+
+## Citation
+
+```bibtex
+@misc{tesslate_synthias127b,
+  title={Synthia-S1-27b: Advanced Reasoning and Coding Model},
+  author={tesslate},
+  year={2025},
+  publisher={tesslate},
+  url={https://tesslate.com}
+}
+```
+
+**Developed by Tesslate** **[Huggingface](https://huggingface.co/tesslate)** **|** **[Website](https://tesslate.com)**
+[Image Source](https://pixabay.com/illustrations/girl-backpack-night-surreal-sky-8257551/)

added_tokens.json

@@ -0,0 +1 @@
+  {"<image_soft_token>": 262144} 

chat_template.json

@@ -0,0 +1 @@
+ {"chat_template": "{{ bos_token }}\n{%- if messages[0]['role'] == 'system' -%}\n    {%- if messages[0]['content'] is string -%}\n        {%- set first_user_prefix = messages[0]['content'] + '\n\n' -%}\n    {%- else -%}\n        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '\n\n' -%}\n    {%- endif -%}\n    {%- set loop_messages = messages[1:] -%}\n{%- else -%}\n    {%- set first_user_prefix = \"\" -%}\n    {%- set loop_messages = messages -%}\n{%- endif -%}\n{%- for message in loop_messages -%}\n    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}\n        {{ raise_exception(\"Conversation roles must alternate user/assistant/user/assistant/...\") }}\n    {%- endif -%}\n    {%- if (message['role'] == 'assistant') -%}\n        {%- set role = \"model\" -%}\n    {%- else -%}\n        {%- set role = message['role'] -%}\n    {%- endif -%}\n    {{ '<start_of_turn>' + role + '\n' + (first_user_prefix if loop.first else \"\") }}\n    {%- if message['content'] is string -%}\n        {{ message['content'] | trim }}\n    {%- elif message['content'] is iterable -%}\n        {%- for item in message['content'] -%}\n            {%- if item['type'] == 'image' -%}\n                {{ '<start_of_image>' }}\n            {%- elif item['type'] == 'text' -%}\n                {{ item['text'] | trim }}\n            {%- endif -%}\n        {%- endfor -%}\n    {%- else -%}\n        {{ raise_exception(\"Invalid content type\") }}\n    {%- endif -%}\n    {{ '<end_of_turn>\n' }}\n{%- endfor -%}\n{%- if add_generation_prompt -%}\n    {{'<start_of_turn>model\n'}}\n{%- endif -%}\n"}    

config.json

@@ -0,0 +1 @@
+  {"architectures": ["Gemma3ForConditionalGeneration"], "boi_token_index": 255999, "eoi_token_index": 256000, "eos_token_id": [1, 106], "image_token_index": 262144, "initializer_range": 0.02, "mm_tokens_per_image": 256, "model_type": "gemma3", "text_config": {"head_dim": 128, "hidden_size": 5376, "intermediate_size": 21504, "model_type": "gemma3_text", "num_attention_heads": 32, "num_hidden_layers": 62, "num_key_value_heads": 16, "query_pre_attn_scalar": 168, "rope_scaling": {"factor": 8.0, "rope_type": "linear"}, "sliding_window": 1024}, "torch_dtype": "bfloat16", "transformers_version": "4.50.0.dev0", "vision_config": {"hidden_size": 1152, "image_size": 896, "intermediate_size": 4304, "model_type": "siglip_vision_model", "num_attention_heads": 16, "num_hidden_layers": 27, "patch_size": 14, "vision_use_head": false}}   

generation_config.json

@@ -0,0 +1 @@
+{"bos_token_id": 2, "cache_implementation": "hybrid", "do_sample": true, "eos_token_id": [1, 106], "pad_token_id": 0, "top_k": 64, "top_p": 0.95, "transformers_version": "4.50.0.dev0"}         

model-00001-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd00d2c14a8ce5b8d356b425c514612903f6aa527b5fb7e660ff9c2c8a6dd0e3
+size 4854573768

model-00002-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d799b58271450e592254820316856c4da60370059b02ab62f9bf1e8662d0bac7
+size 4954793008

model-00003-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:20491acaa3f71229ab2742be973b88badd0ed0dd6d795c09697d5f2b50d18c7c
+size 4954793048

model-00004-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:15138bf0ba5a4f8c5f48ece5cd98ea549ed9df9f2653dfd1b31de9bc1fbda642
+size 4954793088

model-00005-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd8265781f5374eddca2c9831bdaa86081a0515158c5d67ba5bf9e3183e6e620
+size 4954793088

model-00006-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:475d4cc895aaf1c3d54076d59d040c989224615a3883539fde2940fa374ee866
+size 4954793088

model-00007-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:35bd5c886d51c18c0a86ede6096f9898b4e5081ca19f79f5b4ecb34725e5b9ca
+size 4954793088

model-00008-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a813200ce5d057eb048298e29eca82962275c89241629b3f4666cee10430b8f
+size 4954793088

model-00009-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3350d3e2bce22cd75c1932ead986d1579e75b2cc0e96f236f05d099cf068426
+size 4954793088

model-00010-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0a2c5a70e9fff2d183a517530e217f72cb05a8a21d17af988b33bbb6d470ce4
+size 4954793088

model-00011-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39f0122cfbf7a219356a8c4f5e7224c817621fb9d69f661f81416ffc5e60c384
+size 4954793088

model-00012-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65200cef7b3f6f838ad8714639f973800946c395dc83205b03ef766469fd4556
+size 462476760

preprocessor_config.json

@@ -0,0 +1 @@
+{"do_convert_rgb": null, "do_normalize": true, "do_pan_and_scan": null, "do_rescale": true, "do_resize": true, "image_mean": [0.5, 0.5, 0.5], "image_processor_type": "Gemma3ImageProcessor", "image_seq_length": 256, "image_std": [0.5, 0.5, 0.5], "pan_and_scan_max_num_crops": null, "pan_and_scan_min_crop_size": null, "pan_and_scan_min_ratio_to_activate": null, "processor_class": "Gemma3Processor", "resample": 2, "rescale_factor": 0.00392156862745098, "size": {"height": 896, "width": 896}}        

processor_config.json

@@ -0,0 +1 @@
+   {"image_seq_length": 256, "processor_class": "Gemma3Processor"}  

special_tokens_map.json

@@ -0,0 +1 @@
+{"boi_token": "<start_of_image>", "bos_token": {"content": "<bos>", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false}, "eoi_token": "<end_of_image>", "eos_token": {"content": "<eos>", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false}, "image_token": "<image_soft_token>", "pad_token": {"content": "<pad>", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false}, "unk_token": {"content": "<unk>", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false}}   

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a37445c55028d6406e9490d2be970bf316b87ecc5606544f721201ea43c4c6eb
+size 20323114

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
+size 4689074