SmolLM2 - llamafile

The portable/executable AI chatbot for you

Model Summary

This is a portable version of SmolLM2-360M-Instruct with Mozilla/llamafile packing method. It executable, as it is.

Model creator: Hugging Face TB
Original model: HuggingFaceTB/SmolLM2-360M-Instruct

Fabio Matricardi has packaged the LLaMA model into executable weights that we call llamafiles. This gives you the easiest fastest way to use the model on Linux, MacOS, Windows, FreeBSD, OpenBSD 7.3, and NetBSD systems you control on both AMD64 and ARM64.

Software Last Updated: 2025-03-31

Llamafile Version: 0.9.2

The executable will start with a context window set to 8k tokens

About SmolLM2 model family

SmolLM2 is a family of compact language models available in three size: 135M, 360M, and 1.7B parameters. They are capable of solving a wide range of tasks while being lightweight enough to run on-device. More details in our paper: https://arxiv.org/abs/2502.02737

SmolLM2 demonstrates significant advances over its predecessor SmolLM1, particularly in instruction following, knowledge, reasoning. The 360M model was trained on 4 trillion tokens using a diverse dataset combination: FineWeb-Edu, DCLM, The Stack, along with new filtered datasets we curated and will release soon. We developed the instruct version through supervised fine-tuning (SFT) using a combination of public datasets and our own curated datasets. We then applied Direct Preference Optimization (DPO) using UltraFeedback.

The instruct model additionally supports tasks such as text rewriting, summarization and function calling (for the 1.7B) thanks to datasets developed by Argilla such as Synth-APIGen-v0.1. You can find the SFT dataset here: https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk and finetuning code in the alignement handbook

For more details refer to: https://github.com/huggingface/smollm. You will find pre-training, post-training, evaluation and local inference code.

Quickstart

To get started, you need both the LLaMA weights, and the llamafile software. Both of them are included in a single file, which can be downloaded and run as follows:

For Mac/Linux users

wget https://huggingface.co/FM-1976/SmolLM2-360M-it-llamafile/resolve/main/SmolLM2-360M-portable.llamafile
chmod +x SmolLM2-360M-portable.llamafile
./SmolLM2-360M-portable.llamafile

For Windows user:

simply rename the extension from SmolLM2-360M-portable.llamafile to SmolLM2-360M-portable.exe

The default mode of operation for these llamafiles is our new command line chatbot interface. At the same time a Web interface is available at http://127.0.0.1:8080/ and also exposed to your internal Network.

An OpenAI compatible API endpoint server will be listening at http://localhost:8080/v1

Usage

You can use triple quotes to ask questions on multiple lines. You can pass commands like /stats and /context to see runtime status information. You can change the system prompt by passing the -p "new system prompt" flag. You can press CTRL-C to interrupt the model. Finally CTRL-D may be used to exit.

If you prefer to use a web GUI, then a --server mode is provided, that will open a tab with a chatbot and completion interface in your browser. For additional help on how it may be used, pass the --help flag. The server also has an OpenAI API compatible completions endpoint that can be accessed via Python using the openai pip package.

When you launch the executable the oepnAI API server is started automatically

An advanced CLI mode is provided that's useful for shell scripting. You can use it by passing the --cli flag. For additional help on how it may be used, pass the --help flag.

details of the executsble packaging

I used the following .args file

-m
SmolLM2-360M-Instruct.Q8_0.gguf
--host
0.0.0.0
-c
8192
...

You need to download the official binaries from Mozilla/llamafile to do it yourself:

llamafile-0.9.2.exe
zipalign-0.9.2.exe

Additionally you need the weights quiantized of the model. I used the Bartowski repository ones:

file name: SmolLM2-360M-Instruct-Q8_0.gguf
Link: https://huggingface.co/bartowski/SmolLM2-360M-Instruct-GGUF/resolve/main/SmolLM2-360M-Instruct-Q8_0.gguf

Prompt format

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Quickstart with python and openAI API endpoint

Here provides a code snippet with apply_chat_template to show you how to load the tokenizer and model and how to generate contents.

# Chat with an intelligent assistant in your terminal
from openai import OpenAI
import sys

# Point to the local server
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
history = [
    {"role": "system", "content": "You are SMOLLM2-PORTABLE, an intelligent assistant. You always provide well-reasoned answers that are both correct and helpful. Always reply in the language of the instructions."},
    {"role": "user", "content": "Hello, introduce yourself to someone opening this program for the first time. Be concise."},
]
print("\033[92;1m")
while True:
    userinput = ""
    completion = client.chat.completions.create(
        model="local-model", # this field is currently unused
        messages=history,
        temperature=0.3,
        frequency_penalty  = 1.4,
        max_tokens = 600,
        stream=True,
    )

    new_message = {"role": "assistant", "content": ""}
    
    for chunk in completion:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="", flush=True)
            new_message["content"] += chunk.choices[0].delta.content

    history.append(new_message)

    print("\033[1;30m")  #dark grey
    print("Enter your text (end input with Ctrl+D on Unix or Ctrl+Z on Windows) - type quit! to exit the chatroom:")
    print("\033[91;1m")  #red
    lines = sys.stdin.readlines()
    for line in lines:
        userinput += line + "\n"
    if "quit!" in lines[0].lower():
        print("\033[0mBYE BYE!")
        break
    history = [
            {"role": "system", "content": "You are an intelligent assistant. You always provide well-reasoned answers that are both correct and helpful."},
            ]
    history.append({"role": "user", "content": userinput})
    print("\033[92;1m")

Context Window

This model has a max context window size of 8k tokens. By default, a context window size of 8192 tokens is used. You can ask llamafile to use the maximum context size by passing the -c 0 flag. That's big enough for a small book. If you want to be able to have a conversation with your book, you can use the -f book.txt flag.

Evaluation

In this section, we report the evaluation results of SmolLM2. All evaluations are zero-shot unless stated otherwise, and we use lighteval to run them.

Base Pre-Trained Model

Metrics	SmolLM2-360M	Qwen2.5-0.5B	SmolLM-360M
HellaSwag	54.5	51.2	51.8
ARC (Average)	53.0	45.4	50.1
PIQA	71.7	69.9	71.6
MMLU (cloze)	35.8	33.7	34.4
CommonsenseQA	38.0	31.6	35.3
TriviaQA	16.9	4.3	9.1
Winogrande	52.5	54.1	52.8
OpenBookQA	37.4	37.4	37.2
GSM8K (5-shot)	3.2	33.4	1.6

Instruction Model

Metric	SmolLM2-360M-Instruct	Qwen2.5-0.5B-Instruct	SmolLM-360M-Instruct
IFEval (Average prompt/inst)	41.0	31.6	19.8
MT-Bench	3.66	4.16	3.37
HellaSwag	52.1	48.0	47.9
ARC (Average)	43.7	37.3	38.8
PIQA	70.8	67.2	69.4
MMLU (cloze)	32.8	31.7	30.6
BBH (3-shot)	27.3	30.7	24.4
GSM8K (5-shot)	7.43	26.8	1.36

Limitations

SmolLM2 models primarily understand and generate content in English. They can produce text on a variety of topics, but the generated content may not always be factually accurate, logically consistent, or free from biases present in the training data. These models should be used as assistive tools rather than definitive sources of information. Users should always verify important information and critically evaluate any generated content.

Training

Model

Architecture: Transformer decoder
Pretraining tokens: 4T
Precision: bfloat16

Hardware

GPUs: 64 H100

Software

Training Framework: nanotron

License

Apache 2.0

Citation

@misc{allal2025smollm2smolgoesbig,
      title={SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model}, 
      author={Loubna Ben Allal and Anton Lozhkov and Elie Bakouch and Gabriel Martín Blázquez and Guilherme Penedo and Lewis Tunstall and Andrés Marafioti and Hynek Kydlíček and Agustín Piqueres Lajarín and Vaibhav Srivastav and Joshua Lochner and Caleb Fahlgren and Xuan-Son Nguyen and Clémentine Fourrier and Ben Burtenshaw and Hugo Larcher and Haojun Zhao and Cyril Zakka and Mathieu Morlon and Colin Raffel and Leandro von Werra and Thomas Wolf},
      year={2025},
      eprint={2502.02737},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.02737}, 
}

FM-1976
/

SmolLM2-360M-it-llamafile