---
license: mit
datasets:
- tiiuae/falcon-refinedweb
- HuggingFaceFW/fineweb
base_model:
  - cckm/tinymistral_950m
language:
- en
pipeline_tag: text-generation
library_name: PyTorch
---

## A deep and narrow Mistral model (950M params)
This checkpoint is for a small (950M params), deep and narrow (40 layers, hidden size=1440) Mistral model, as described in this [[blog post]](https://epsilons.ai/blog.html#post1_3). It is meant for edge applications.

It was trained with ~400B tokens from RefinedWeb, and ~400B tokens from FineWeb (up to epoch 202418). It is a base model, and has not gone through instruct or chat fine-tuning.

LM Harness numbers:
| Benchmark | Result |
| ----- | ----- |
| arc_c | 0.2884 |
| arc_e | 0.5139 |
| boolq | 0.6089 |
| hellaswag | 0.5888 |
| obqa | 0.3280 |
| piqa | 0.7388 |
| siqa | 0.4038 |
| wino | 0.5627 |