Lil Bard historical checkpoints
This repository contains 25 loadable Transformers checkpoints from the
pretraining run for N8Programs/lil-bard.
The final step-25,485 model is kept in that separate repository so ordinary
users do not download the full checkpoint history.
Checkpoints span step 1,000 through step 25,000 in increments of 1,000. Each
subfolder contains a complete Qwen3MoeForCausalLM export and tokenizer.
Loading a checkpoint
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "N8Programs/lil-bard-checkpts"
subfolder = "step-0001000"
tokenizer = AutoTokenizer.from_pretrained(repo_id, subfolder=subfolder)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
subfolder=subfolder,
dtype=torch.bfloat16,
device_map="auto",
)
Available subfolders are step-0001000, step-0002000, ..., through
step-0025000.
These historical exports retain the fused expert-tensor layout used during the
run. They are tested with Transformers 5.11.0. Transformers 4.57.1 users should
use the cross-version final checkpoint in N8Programs/lil-bard.
Run summary
Lil Bard is a 172,052,992-parameter English story MoE with 58,806,784 active parameters per token, 8 experts with top-2 routing, and an 8,192-entry whole-document byte-level BPE tokenizer. It was pretrained from scratch for 2,492,032,616 real loss tokens on two NVIDIA GB10 systems.
The final model, architecture and data details, evaluations, usage example,
and limitations are documented in the
N8Programs/lil-bard model card.
The complete training trace is available in the
lil_bard_moe_8x2_full W&B run.
Only model exports are included here. The larger optimizer/scheduler resume checkpoints are not uploaded.