README.md

---
license: mit
datasets:
- pfb30/multi_woz_v22
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- Sam-2
- text-generation
---

# 🧠 Model Card: Sam‑2.0

## 📌 Model Overview
**Sam‑2.0** is a minimal, modular, decoder‑only Transformer architecture designed for chat‑style reasoning tasks.  
It emphasizes reproducibility, ablation‑friendly design, and clean benchmarking across input modalities.

- **Architecture**: Decoder‑only Transformer with RMSNorm, SwiGLU feed‑forward, and causal masking  
- **Training Objective**: Causal language modeling (CLM) with role‑based label masking  
- **Checkpoint**: `sam2-epoch35.safetensors`  
- **Final Train Loss**: 1.04  
- **Validation Loss**: Not tracked in this run  
- **Training Duration**: ~6272 s over 35 epochs  
- **Framework**: PyTorch + Hugging Face Transformers (custom model class)

## 🧱 Model Architecture
| Component         | Description                                                                 |
|-------------------|-----------------------------------------------------------------------------|
| Backbone          | Decoder‑only Transformer stack                                              |
| Normalization     | RMSNorm                                                                     |
| Attention         | Multi‑head self‑attention (causal)                                          |
| Feed‑Forward      | SwiGLU activation with dropout                                              |
| Positional Bias   | Learned absolute positions (no RoPE in this minimal variant)                |
| Head              | Tied‑embedding LM head                                                      |
| Checkpoint Format | `safetensors` with metadata for reproducibility                             |

## 🧪 Training Details
- **Dataset**: [pfb30/multi_woz_v22](https://huggingface.co/datasets/pfb30/multi_woz_v22)  
- **Batch Size**: 8  
- **Optimizer**: AdamW  
- **Learning Rate**: 2 × 10⁻⁴ (constant in this run)  
- **Loss Function**: Cross‑entropy over assistant tokens only  
- **Hardware**: Kaggle GPU runtime  
- **Logging**: Step‑wise loss tracking, no validation during training

## 📊 Evaluation
| Metric           | Value       | Notes                                 |
|------------------|-------------|---------------------------------------|
| Final Train Loss | 1.04        | Achieved at Epoch 35/35               |
| Validation Loss  | —           | Not tracked in this run               |
| Inference Speed  | Fast        | Lightweight architecture              |
| Generalisation   | TBD         | To be compared against Sam‑2.5        |

## 🔧 Intended Use
- **Research**: Benchmarking modular architectures and ablation studies  
- **Education**: Reasoning scaffolds and logic quizzes  
- **Deployment**: Lightweight agents for chat and dialogue modeling

## 🚫 Limitations
- No validation tracking — generalisation must be inferred via external harnesses  
- Trained on MultiWOZ v2.2 only — may not generalize to other domains without fine‑tuning  
- Minimal architecture — no RoPE/MQA in this variant

## 📁 Files
- `sam2-epoch35.safetensors` — final checkpoint  
- `config.json` — architecture and training config  
- `tokenizer.json` — tokenizer with special tokens  
- `README.md` — training logs and setup instructions

## 🧩 How to Load
```python
from transformers import AutoTokenizer
import torch
from sam2 import Sam2, Sam2Config  # your custom model class

tok = AutoTokenizer.from_pretrained("Smilyai-labs/Sam-2.0")
cfg = Sam2Config(**json.load(open("config.json")))
model = Sam2(cfg)
state = torch.load("sam2-epoch35.safetensors", map_location="cpu")
model.load_state_dict(state)
model.eval()

prompt = "<|user|> Hello! <|eot|>\n<|assistant|>"
ids = tok.encode(prompt, return_tensors="pt")
with torch.no_grad():
    for _ in range(50):
        logits = model(ids)
        next_id = torch.argmax(logits[:, -1, :], dim=-1, keepdim=True)
        ids = torch.cat([ids, next_id], dim=1)
        if next_id.item() == tok.eos_token_id:
            break

print(tok.decode(ids[0]))