README.md

---
library_name: transformers
tags:
- mamba
- deepseek
- reasoning
base_model:
- tiiuae/Falcon3-Mamba-7B-Instruct
pipeline_tag: text-generation
---

# Falcon3-Mamba-R1-v0
<img src="https://i.ibb.co/8DbzbXXC/Untitled-design.png" width="360" height="360" />

## Model Details

**Model Description:**

This model is a fine-tuned version of Falcon3-Mamba-7B-Instruct, optimized for logical reasoning and structured problem-solving before generating responses.  

It leverages the Mamba architecture, which scales linearly with an increased number of tokens, making it an efficient and fast reasoning model while maintaining high response quality. 

This fine-tuned version comes from an earlier checkpoint of the fine tuning pipeline.

*   **Developed by:** Hanzla Javaid
*   **Base Model:** tiiuae/Falcon3-Mamba-7B-Instruct
*   **Model Type:** Mamba-based causal decoder
*   **Model Release Date:** March 2025

## Intended Uses

**Direct Use:**

This model is designed for:

*   Reasoning-heavy tasks (math, logic, and structured problem-solving)
*   STEM-based question-answering
*   General-purpose text generation

**Downstream Use:**

*   Fine-tuning for domain-specific applications such as finance, law, medicine, and research.
*   Integration into chatbots and virtual assistants that require advanced reasoning skills.
*   Enhancement of automated coding assistants with structured logic building.

**Out-of-Scope Use:**

*   Misinformation or deceptive applications
*   Automated decision-making in high-risk fields (e.g., medical diagnosis without human oversight)
*   Bias-sensitive applications where fairness is critical but not explicitly controlled

## Bias and Limitations

**Known Biases:**

*   The model prioritizes English language data, so performance on multilingual tasks may be weaker.
*   Fine-tuning may introduce or amplify biases present in the training data, especially in areas like ethics, politics, and cultural perspectives.

**Technical Limitations:**

*   Performance may degrade on long-form generation beyond 64K tokens.


**Recommendations:**

*   Users should verify outputs for accuracy, especially in critical applications.
*   Regular bias evaluation should be conducted when deploying in production environments.

## Getting Started

To use this model, you can load it with transformers:

```python
repo_name = "hanzla/Falcon3-Mamba-R1-v0"
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained(repo_name)

model = AutoModelForCausalLM.from_pretrained(
    repo_name,
    device_map="auto",
    torch_dtype=torch.float16,
)

def generate_text(prompt,generation_model,generation_tokenizer,max_tokens=1024):
    messages = [
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": prompt},
    ]
    input_text = generation_tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    print(input_text)
    input_ids = generation_tokenizer(input_text, return_tensors="pt").input_ids.to("auto")
    outputs = generation_model.generate(input_ids, max_new_tokens=max_tokens)
    generated_tokens = outputs[0][len(input_ids[0]):] 
    return tokenizer.decode(generated_tokens, skip_special_tokens=True)
    
```

## Training Details


**Training Procedure:**

*   **Pretrained Base Model:** Falcon3-Mamba-7B-Instruct
*   **Fine-tuning Data:** A subset of STEM problems from open-thoughts/OpenThoughts-114k
*   **Training Strategy:** GRPO
*   **Training Hyperparameters:**
    *   **Batch Size:** 4
    *   **Epochs:** 3
    *   **Precision:** Mixed (fp16 / bf16)
    *   **Hardware:** 2xH100 GPUs

## Evaluation

**Testing Data and Metrics:**

The fine-tuned model's performance was evaluated on a variety of benchmarks to assess its reasoning abilities and overall capabilities.  The table below presents a comparison between the fine-tuned model and the base model:

| Category      | Benchmark                      | Falcon3-Mamba-R1-v0  | Base Falcon3-Mamba-7B-Instruct |
|---------------|--------------------------------|----------------------------------------|---------------------------------|
| General       | MMLU (5-shot)                  | 72.1                                   | 65.3                            |
| Math          | GSM8K (5-shot)                 | 89.5                                   | 65.2                            |


## Technical Specifications

**Model Architecture:**

*   **Mamba Blocks:** 64
*   **Hidden Size:** 4096

**Software Requirements:**

*   `transformers >= 4.38`
*   `torch >= 2.1`
*   `accelerate >= 0.25`
*   `mamba-ssm`
*   `causal-conv1d>=1.4.0`