|
--- |
|
license: mit |
|
datasets: |
|
- tiiuae/falcon-refinedweb |
|
- HuggingFaceFW/fineweb |
|
base_model: |
|
- cckm/tinymistral_950m |
|
language: |
|
- en |
|
pipeline_tag: text-generation |
|
library_name: PyTorch |
|
--- |
|
|
|
## A deep and narrow Mistral model (950M params) |
|
This checkpoint is for a small (950M params), deep and narrow (40 layers, hidden size=1440) Mistral model, as described in this [[blog post]](https://epsilons.ai/blog.html#post1_3). It is meant for edge applications. |
|
|
|
It was trained with ~400B tokens from RefinedWeb, and ~400B tokens from FineWeb (up to epoch 202418). It is a base model, and has not gone through instruct or chat fine-tuning. |
|
|
|
LM Harness numbers: |
|
| Benchmark | Result | |
|
| ----- | ----- | |
|
| arc_c | 0.2884 | |
|
| arc_e | 0.5139 | |
|
| boolq | 0.6089 | |
|
| hellaswag | 0.5888 | |
|
| obqa | 0.3280 | |
|
| piqa | 0.7388 | |
|
| siqa | 0.4038 | |
|
| wino | 0.5627 | |
