See axolotl config

axolotl version: 0.12.2

base_model: google/gemma-3n-E2B-it
hub_model_id: sudoping01/bambara-llm-exp3 
plugins:
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
cut_cross_entropy: true
load_in_4bit: false  
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
ddp: true
chat_template: gemma3n
eot_tokens:
  - <end_of_turn>
special_tokens:
  eot_token: <end_of_turn>
datasets:
  - path: sudoping01/bambara-instructions
    type: chat_template
    split: train
    name: cleaned
    field_messages: messages
    message_property_mappings:
      role: role
      content: content
val_set_size: 0.01
output_dir: ./outputs/bambara-gemma3n-lora-exp4
adapter: lora  
lora_r: 64     
lora_alpha: 128 
lora_dropout: 0.05
lora_target_modules: 'model.language_model.layers.[\d]+.(mlp|self_attn).(up|down|gate|q|k|v|o)_proj'
sequence_len: 4096 
sample_packing: false
pad_to_sequence_len: false
micro_batch_size: 8  
gradient_accumulation_steps: 2
num_epochs: 3  
optimizer: adamw_8bit
lr_scheduler: cosine
learning_rate: 1.2e-4  
warmup_ratio: 0.03
weight_decay: 0.01
bf16: auto
tf32: false
logging_steps: 10
saves_per_epoch: 2  
evals_per_epoch: 2

bambara-llm-exp3

This model is a fine-tuned version of google/gemma-3n-E2B-it on the sudoping01/bambara-instructions dataset. It achieves the following results on the evaluation set:

• Loss: 0.4952
• Memory/max Mem Active(gib): 57.85
• Memory/max Mem Allocated(gib): 57.85
• Memory/device Mem Reserved(gib): 59.82

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.00012
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 2
• total_train_batch_size: 128
• total_eval_batch_size: 64
• optimizer: Use adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 633
• training_steps: 21126

Training results

Training Loss	Epoch	Step	Validation Loss	Mem Active(gib)	Mem Allocated(gib)	Mem Reserved(gib)
No log	0	0	7.4595	19.86	19.86	20.35
0.8265	0.5	3521	0.7787	57.85	57.85	59.82
0.7107	1.0	7042	0.6745	57.85	57.85	59.82
0.6363	1.5	10563	0.6026	57.85	57.85	59.82
0.5421	2.0	14084	0.5429	57.85	57.85	59.82
0.5733	2.5	17605	0.5039	57.85	57.85	59.82
0.5401	3.0	21126	0.4952	57.85	57.85	59.82

Framework versions

• PEFT 0.17.0
• Transformers 4.55.2
• Pytorch 2.6.0+cu124
• Datasets 4.0.0
• Tokenizers 0.21.4