Sommerfugl-31B

Sommerfugl ("butterfly") is a family of Norwegian language models by oolabs.no.

Built with Gemma. This model is a finetune of google/gemma-4-31B-it, modified by oolabs.no. Use is governed by the Gemma Terms of Use.

Why

Google's instruction tuning of Gemma 4 damages its Norwegian: on the National Library of Norway's leaderboard, gemma-4-31B-it ranks in the top 3 on eleven categories but collapses to ~rank 48 on Norwegian language knowledge and ~rank 51 on Norwegian summarization. Sommerfugl-31B repairs both while retaining the base's strengths.

Results

Evaluated under the National Library's published leaderboard protocol (lm-evaluation-harness + NorEval; cloze MC, chat template, best over prompt versions × {0,5}-shot, greedy decoding). Gemma column: NB's own leaderboard row. Sommerfugl and Borealis 2 (NbAiLab's newest flagship) evaluated by us under the identical protocol.

task (metric)	gemma-4-31B-it	Sommerfugl-31B	Borealis 2 preview
Language knowledge: NoCoLA (acc)	0.828	0.859	0.834
Language knowledge: NCB (acc)	0.755	0.825	0.774
Summaries: NorSumm nob / nno (bleu)	4.08 / 3.37	7.39 / 7.00	6.62 / 5.15
Summarize on request (bleu)	0.98	4.61	1.75
Rewrite (bleu)	0.03	3.66	0.11
Idioms nob / nno (fscore)	0.085 / 0.141	0.442 / 0.541	0.121 / 0.160
Grammar correction (exact match)	0.294	0.402	0.342
Reading comp.: NorQuAD (f1)	0.445	0.748	0.799
Translation en↔nb/nn (bleu, 4-dir avg)	57.2	59.1	58.1
Belebele (acc)	0.938	0.938	0.904
OpenbookQA / CommonsenseQA (acc)	0.968 / 0.860	0.957 / 0.830	0.955 / 0.808
NoReC sentiment (acc)	0.910	0.923	0.918
TruthfulQA mc nob / nno (acc)	0.850 / 0.930	0.664 / 0.772	0.811 / 0.842
MMLU English (acc)	0.831	0.831	0.550
MMLU Norwegian (acc)	0.859	0.824	—

Head-to-head vs Borealis 2 across all 21 protocol-complete rows: 18–3.

Known limitations (reported deliberately)

• TruthfulQA (mc) regresses vs the base -it (−0.19): supervised finetuning erodes part of the base's RLHF truthfulness calibration.
• MMLU-nb −0.035 vs the base -it.
• Not evaluated: NB's unpublished internal tasks (translated ARC/GSM8K/IFEval/GoldenSwag, safety sets — ~8% of their suite) and the heavy 5-shot MC variants; exclusions applied identically to every model we compare.
• We publish no overall/aggregate score: that number belongs to the National Library's own evaluation pipeline.

Evaluation integrity

Training data was quality-gated and decontaminated against all 19 Norwegian evaluation datasets used above (all splits), so the reported numbers are not inflated by train/test overlap.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("oolabs/sommerfugl-31b")
model = AutoModelForCausalLM.from_pretrained("oolabs/sommerfugl-31b", dtype="bfloat16", device_map="auto")
msgs = [{"role": "user", "content": "Skriv et kort sammendrag av teksten under."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=256, do_sample=False)[0][ids.shape[-1]:]))