|
|
--- |
|
|
license: apache-2.0 |
|
|
base_model: |
|
|
- Writer/palmyra-mini-thinking-a |
|
|
tags: |
|
|
- mlx |
|
|
- qwen2 |
|
|
- palmyra |
|
|
- thinking |
|
|
- reasoning |
|
|
--- |
|
|
|
|
|
# Palmyra Mini Thinking A - MLX BF16 |
|
|
|
|
|
## Model Description |
|
|
|
|
|
This is a bfloat16 precision version of the [palmyra-mini-thinking-a model](https://huggingface.co/Writer/palmyra-mini-thinking-a), optimized for Apple Silicon using the MLX framework. This model is based on the Qwen2 architecture and is specifically designed for reasoning tasks with explicit thinking capabilities through special `<think>` and `</think>` tokens. |
|
|
|
|
|
## Quick Start |
|
|
|
|
|
### Installation |
|
|
|
|
|
```bash |
|
|
pip install mlx-lm |
|
|
``` |
|
|
|
|
|
### Usage |
|
|
|
|
|
```python |
|
|
from mlx_lm import load, generate |
|
|
|
|
|
# Load the model |
|
|
model, tokenizer = load("/Users/thomas/Documents/Model Weights/SPW2 Mini Launch/palmyra-mini-thinking-a/MLX") |
|
|
|
|
|
# Generate text with thinking |
|
|
prompt = "Solve this step by step: What is 15% of 240?" |
|
|
response = generate(model, tokenizer, prompt=prompt, verbose=True, max_tokens=512) |
|
|
print(response) |
|
|
``` |
|
|
|
|
|
## Technical Specifications |
|
|
|
|
|
### Model Architecture |
|
|
- **Model Type**: `qwen2` (Qwen2 Architecture) |
|
|
- **Architecture**: `Qwen2ForCausalLM` |
|
|
- **Parameters**: ~1.7 billion parameters |
|
|
- **Precision**: bfloat16 |
|
|
- **Specialization**: Reasoning and thinking tasks |
|
|
|
|
|
### Core Parameters |
|
|
| Parameter | Value | |
|
|
|-----------|-------| |
|
|
| Hidden Size | 1,536 | |
|
|
| Intermediate Size | 8,960 | |
|
|
| Number of Layers | 28 | |
|
|
| Attention Heads | 12 | |
|
|
| Key-Value Heads | 2 | |
|
|
| Head Dimension | 128 | |
|
|
| Vocabulary Size | 151,665 | |
|
|
|
|
|
### Attention Mechanism |
|
|
- **Attention Type**: Full attention across all 28 layers |
|
|
- **Max Position Embeddings**: 131,072 tokens |
|
|
- **Attention Dropout**: 0.0 |
|
|
- **Sliding Window**: Not used |
|
|
- **Max Window Layers**: 21 |
|
|
|
|
|
### RoPE (Rotary Position Embedding) Configuration |
|
|
- **RoPE Theta**: 10,000 |
|
|
- **RoPE Scaling**: None |
|
|
|
|
|
### Thinking Capabilities |
|
|
- **Thinking Tokens**: `<think>` (151648) and `</think>` (151649) |
|
|
- **Reasoning Mode**: Explicit step-by-step reasoning |
|
|
- **Chat Template**: Automatically adds `<think>` tag for generation prompts |
|
|
|
|
|
### File Structure |
|
|
``` |
|
|
palmyra-mini-thinking-a/MLX/ |
|
|
├── config.json # Model configuration |
|
|
├── model.safetensors # Model weights (3.3GB) |
|
|
├── model.safetensors.index.json # Model sharding index |
|
|
├── tokenizer.json # Tokenizer configuration |
|
|
├── tokenizer_config.json # Tokenizer settings |
|
|
├── special_tokens_map.json # Special tokens mapping |
|
|
├── chat_template.jinja # Chat template with thinking |
|
|
└── README.md # Model documentation |
|
|
``` |
|
|
|
|
|
## Performance Characteristics |
|
|
|
|
|
### Hardware Requirements |
|
|
- **Platform**: Apple Silicon (M1, M2, M3, M4 series) |
|
|
- **Memory**: ~3.3GB for model weights |
|
|
- **Recommended RAM**: 12GB+ for optimal performance |
|
|
- **Precision**: Full bfloat16 precision |
|
|
|
|
|
### Layer Configuration |
|
|
All 28 layers use full attention mechanism as specified in the `layer_types` configuration, providing consistent attention patterns across the entire model depth. |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Tokenizer |
|
|
- **Type**: LlamaTokenizerFast with 151,665 vocabulary size |
|
|
- **Special Tokens**: |
|
|
- BOS Token ID: 151646 (` |
|
|
`) |
|
|
- EOS Token ID: 151643 (` |
|
|
`) |
|
|
- Pad Token ID: 151643 (` |
|
|
`) |
|
|
- Think Start: 151648 (`<think>`) |
|
|
- Think End: 151649 (`</think>`) |
|
|
|
|
|
### Model Configuration |
|
|
- **Hidden Activation**: SiLU (Swish) |
|
|
- **Normalization**: RMSNorm (ε = 1e-06) |
|
|
- **Initializer Range**: 0.02 |
|
|
- **Attention Dropout**: 0.0 |
|
|
- **Word Embeddings**: Not tied |
|
|
- **Use Cache**: False (optimized for thinking tasks) |
|
|
|
|
|
### Chat Template |
|
|
The model uses a specialized chat template that automatically initiates thinking mode: |
|
|
- User messages: ` |
|
|
` |
|
|
- Assistant messages: ` |
|
|
<｜Assistant｜><think>\n` (automatically adds thinking prompt) |
|
|
- Tool calling support with `<tool_call>` and `</tool_call>` tokens |
|
|
- Vision and multimodal tokens included |
|
|
|
|
|
## Usage Examples |
|
|
|
|
|
### Reasoning Task |
|
|
```python |
|
|
prompt = """ |
|
|
A train travels 120 miles in 2 hours. If it maintains the same speed, how far will it travel in 5 hours? |
|
|
<｜Assistant｜><think> |
|
|
""" |
|
|
|
|
|
response = generate(model, tokenizer, prompt=prompt, max_tokens=300) |
|
|
``` |
|
|
|
|
|
### Problem Solving |
|
|
```python |
|
|
prompt = """ |
|
|
Explain why the sky appears blue during the day. |
|
|
<｜Assistant｜><think> |
|
|
""" |
|
|
|
|
|
response = generate(model, tokenizer, prompt=prompt, max_tokens=400) |
|
|
``` |
|
|
|
|
|
## Known Limitations |
|
|
|
|
|
1. **Platform Dependency**: Optimized specifically for Apple Silicon; may not run on other platforms |
|
|
2. **Memory Requirements**: Requires significant memory due to full precision weights |
|
|
3. **Thinking Overhead**: Explicit thinking may increase response length and generation time |
|
|
4. **Cache Disabled**: Model has `use_cache: false` which may impact inference speed |
|
|
|
|
|
## Compatibility |
|
|
|
|
|
- **MLX-LM**: Requires recent version with Qwen2 support |
|
|
- **Apple Silicon**: M1, M2, M3, M4 series processors |
|
|
- **macOS**: Compatible with recent macOS versions supporting MLX |
|
|
- **Transformers**: Version 4.52.4+ |
|
|
|
|
|
## License |
|
|
|
|
|
Apache 2.0 |
|
|
|
|
|
#### Original model card below: |
|
|
|
|
|
------ |
|
|
|
|
|
|
|
|
<div align="center"> |
|
|
<h1>Palmyra-mini-thinking-a</h1> |
|
|
|
|
|
</div> |
|
|
|
|
|
### Model Description |
|
|
|
|
|
- **Language(s) (NLP):** English |
|
|
- **License:** Apache-2.0 |
|
|
- **Finetuned from model:** Qwen/Qwen2.5-1.5B |
|
|
- **Context window:** 131,072 tokens |
|
|
- **Parameters:** 1.7 billion |
|
|
|
|
|
|
|
|
## Model Details |
|
|
|
|
|
The palmyra-mini-thinking-a model demonstrates exceptional performance in advanced mathematical reasoning and competitive programming. Its capabilities are highlighted by an outstanding score of 0.886 on the 'MATH500' benchmark, showcasing a robust ability to solve complex mathematical problems. The strength of the model in quantitative challenges is further confirmed by its score of 0.8287 on 'gsm8k (strict-match)', which demonstrates proficiency in multi-step arithmetic reasoning. Additionally, the model proves its aptitude for high-level problem-solving with a score of 0.8 on 'AMC23'. The model also shows strong potential in the coding domain, achieving a score of 0.5631 on 'Codeforces (pass_rate)' and 0.5481 on 'Olympiadbench (extractive_match)', indicating competence in generating correct solutions for programming challenges. |
|
|
|
|
|
## Benchmark Performance |
|
|
|
|
|
This section provides a detailed breakdown of the palmyra-mini-thinking-a model's performance across a standardized set of industry benchmarks. The data is presented in its original order from the source evaluation. |
|
|
|
|
|
| Benchmark | Score | |
|
|
|:-----------------------------------------------------------------|---------:| |
|
|
| gsm8k (strict-match) | 0.8287 | |
|
|
| minerva_math(exact_match) | 0.3842 | |
|
|
| mmlu_pro(exact_match) | 0.2748 | |
|
|
| hendrycks_math | 0.0054 | |
|
|
| ifeval (inst_level_loose_acc) | 0.3657 | |
|
|
| mathqa (acc) | 0.4171 | |
|
|
| humaneval (pass@1) | 0.2378 | |
|
|
| BBH (get-answer)(exact_match) | 0.462 | |
|
|
| mbpp | 0.304 | |
|
|
| leadboard_musr (acc_norm) | 0.3413 | |
|
|
| gpqa lighteval gpqa diamond_pass@1:8_samples | 0.3826 | |
|
|
| AIME24(pass@1)(avg-of-1) | 0.4333 | |
|
|
| AIME25(pass@1)(avg-of-1) | 0.3667 | |
|
|
| Livecodebench-codegen (livecodebench/code_generation_lite v4_v5) | 0.1784 | |
|
|
| AMC23 | 0.8 | |
|
|
| MATH500 | 0.886 | |
|
|
| Minerva | 0.3493 | |
|
|
| Olympiadbench (extractive_match) | 0.5481 | |
|
|
| Codecontests (pass_rate) | 0.1778 | |
|
|
| Codeforces (pass_rate) | 0.5631 | |
|
|
| Taco (pass_rate) | 0.3083 | |
|
|
| APPS (all_levels) | 0.0447 | |
|
|
| HMMT23 (extractive_match) | 0.1 | |
|
|
| Average | 0.380839 | |
|
|
|
|
|
|
|
|
|
|
|
### Use with transformers |
|
|
|
|
|
You can run conversational inference using the Transformers Auto classes with the `generate()` function. Here's an example: |
|
|
|
|
|
```py |
|
|
import torch |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
|
|
model_id = "Writer/palmyra-mini-thinking-a" |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
|
|
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
model_id, |
|
|
torch_dtype=torch.float16, |
|
|
device_map="auto", |
|
|
attn_implementation="flash_attention_2", |
|
|
) |
|
|
|
|
|
messages = [ |
|
|
{ |
|
|
"role": "user", |
|
|
"content": "You have a 3-liter jug and a 5-liter jug. How can you measure exactly 4 liters of water?" |
|
|
} |
|
|
], |
|
|
|
|
|
input_ids = tokenizer.apply_chat_template( |
|
|
messages, tokenize=True, add_generation_prompt=True, return_tensors="pt" |
|
|
) |
|
|
|
|
|
gen_conf = { |
|
|
"max_new_tokens": 256, |
|
|
"eos_token_id": tokenizer.eos_token_id, |
|
|
"temperature": 0.3, |
|
|
"top_p": 0.9, |
|
|
} |
|
|
|
|
|
with torch.inference_mode(): |
|
|
output_id = model.generate(input_ids, **gen_conf) |
|
|
|
|
|
output_text = tokenizer.decode(output_id[0][input_ids.shape[1] :]) |
|
|
|
|
|
print(output_text) |
|
|
``` |
|
|
|
|
|
## Running with vLLM |
|
|
```py |
|
|
vllm serve Writer/palmyra-mini-thinking-a |
|
|
``` |
|
|
```py |
|
|
curl -X POST http://localhost:8000/v1/chat/completions \ |
|
|
-H "Content-Type: application/json" \ |
|
|
-d '{ |
|
|
"model": "Writer/palmyra-mini-thinking-a", |
|
|
"messages": [ |
|
|
{ |
|
|
"role": "user", |
|
|
"content": "You have a 3-liter jug and a 5-liter jug. How can you measure exactly 4 liters of water?" |
|
|
} |
|
|
], |
|
|
"max_tokens": 8000, |
|
|
"temperature": 0.2 |
|
|
}' |
|
|
``` |
|
|
|
|
|
|
|
|
## Ethical Considerations |
|
|
|
|
|
As with any language model, there is a potential for generating biased or inaccurate information. Users should be aware of these limitations and use the model responsibly. |
|
|
|
|
|
### Citation and Related Information |
|
|
|
|
|
To cite this model: |
|
|
``` |
|
|
@misc{Palmyra-mini-thinking-a, |
|
|
author = {Writer Engineering team}, |
|
|
title = {{Palmyra-mini: A powerful LLM designed for math and coding}}, |
|
|
howpublished = {\url{https://dev.writer.com}}, |
|
|
year = 2025, |
|
|
month = Sep |
|
|
} |
|
|
``` |
|
|
Contact [email protected] |
