---
library_name: transformers
license: apache-2.0
base_model: Dans-DiscountModels/Mistral-Nemo-Base-2407-DanChat
tags:
- axolotl
- generated_from_trainer
datasets:
- Dans-DiscountModels/pretokenization-test-5
model-index:
- name: 12b-mn-dans-personality-engine-v1.3.0-TestArticle-1
results: []
---
[
](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config
axolotl version: `0.10.0.dev0`
```yaml
base_model: Dans-DiscountModels/Mistral-Nemo-Base-2407-DanChat
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code:
# wandb configuration
wandb_project: 12b-mn-dans-personality-engine
wandb_watch:
wandb_run_id: V1.3.0-1-4 # V{Version}-{Run Number}-{Attempt Number}
wandb_log_model:
# push checkpoints to hub
hub_model_id: Dans-DiscountModels/12b-mn-dans-personality-engine-v1.3.0-TestArticle-1
# how to push checkpoints to hub
# https://huggingface.co/docs/transformers/v4.31.0/en/main_classes/trainer#transformers.TrainingArguments.hub_strategy
hub_strategy: "every_save"
# Whether to use hf `use_auth_token` for loading datasets. Useful for fetching private datasets
# Required to be true when used in combination with `push_dataset_to_hub`
hf_use_auth_token: true
# where to save the finished model to
output_dir: ./12b-mn-dans-personality-engine-v1.3.0
# dataset settings (local or huggingface repo)
datasets:
- path: Dans-DiscountModels/pretokenization-test-5
ds_type: parquet
type:
plugins:
- axolotl.integrations.liger.LigerPlugin
- axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
liger_rope: true
liger_rms_norm: true
liger_layer_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true
cut_cross_entropy: true
load_in_8bit: false
load_in_4bit: false
strict: false
adapter:
lora_model_dir:
dataset_prepared_path: ./12b-mn-dans-personality-engine-data
val_set_size: 0.003
sequence_len: 32768
sample_packing: true
eval_sample_packing: true
pad_to_sequence_len: true
gradient_checkpointing: true
gradient_accumulation_steps: 2
micro_batch_size: 2
num_epochs: 2
optimizer: ademamix_8bit
optim_args: "beta1=0.9,beta2=0.999,beta3=0.999,alpha=5"
lr_scheduler: rex
learning_rate: 0.00001
cosine_min_lr_ratio:
weight_decay:
max_grad_norm: 0.001
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false
early_stopping_patience:
resume_from_checkpoint:
auto_resume_from_checkpoints: true
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_ratio: 0.1
evals_per_epoch: 24
eval_table_size:
eval_max_new_tokens:
saves_per_epoch: 2
save_total_limit: 1
debug: false
deepspeed: deepspeed_configs/zero3_bf16.json
fsdp:
fsdp_config:
special_tokens:
```
# 12b-mn-dans-personality-engine-v1.3.0-TestArticle-1
This model is a fine-tuned version of [Dans-DiscountModels/Mistral-Nemo-Base-2407-DanChat](https://huggingface.co/Dans-DiscountModels/Mistral-Nemo-Base-2407-DanChat) on the Dans-DiscountModels/pretokenization-test-5 dataset.
It achieves the following results on the evaluation set:
- Loss: 1.4392
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- total_eval_batch_size: 16
- optimizer: Use ademamix_8bit and the args are:
beta1=0.9,beta2=0.999,beta3=0.999,alpha=5
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 321
- num_epochs: 2.0
### Training results
| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:------:|:----:|:---------------:|
| 1.8086 | 0.0006 | 1 | 1.7459 |
| 1.593 | 0.0417 | 67 | 1.5911 |
| 1.5578 | 0.0833 | 134 | 1.5565 |
| 1.5782 | 0.1250 | 201 | 1.5436 |
| 1.5702 | 0.1666 | 268 | 1.5377 |
| 1.5926 | 0.2083 | 335 | 1.5328 |
| 1.6364 | 0.2499 | 402 | 1.5291 |
| 1.5082 | 0.2916 | 469 | 1.5234 |
| 1.6002 | 0.3332 | 536 | 1.5197 |
| 1.5252 | 0.3749 | 603 | 1.5162 |
| 1.5915 | 0.4165 | 670 | 1.5121 |
| 1.5108 | 0.4582 | 737 | 1.5103 |
| 1.5663 | 0.4998 | 804 | 1.5063 |
| 1.5085 | 0.5415 | 871 | 1.5037 |
| 1.4273 | 0.5832 | 938 | 1.5024 |
| 1.5528 | 0.6248 | 1005 | 1.4994 |
| 1.6072 | 0.6665 | 1072 | 1.4975 |
| 1.6074 | 0.7081 | 1139 | 1.4920 |
| 1.5495 | 0.7498 | 1206 | 1.4904 |
| 1.6117 | 0.7914 | 1273 | 1.4883 |
| 1.4621 | 0.8331 | 1340 | 1.4850 |
| 1.6381 | 0.8747 | 1407 | 1.4838 |
| 1.4221 | 0.9164 | 1474 | 1.4813 |
| 1.5812 | 0.9580 | 1541 | 1.4789 |
| 1.4581 | 0.9997 | 1608 | 1.4750 |
| 1.4608 | 1.0417 | 1675 | 1.4800 |
| 1.5261 | 1.0833 | 1742 | 1.4798 |
| 1.3856 | 1.1250 | 1809 | 1.4796 |
| 1.4469 | 1.1666 | 1876 | 1.4766 |
| 1.4783 | 1.2083 | 1943 | 1.4741 |
| 1.5025 | 1.2499 | 2010 | 1.4733 |
| 1.4531 | 1.2916 | 2077 | 1.4726 |
| 1.4719 | 1.3332 | 2144 | 1.4712 |
| 1.4123 | 1.3749 | 2211 | 1.4700 |
| 1.4653 | 1.4165 | 2278 | 1.4673 |
| 1.4571 | 1.4582 | 2345 | 1.4660 |
| 1.4261 | 1.4998 | 2412 | 1.4660 |
| 1.3212 | 1.5415 | 2479 | 1.4620 |
| 1.3828 | 1.5832 | 2546 | 1.4617 |
| 1.3617 | 1.6248 | 2613 | 1.4597 |
| 1.4364 | 1.6665 | 2680 | 1.4567 |
| 1.4686 | 1.7081 | 2747 | 1.4549 |
| 1.3317 | 1.7498 | 2814 | 1.4530 |
| 1.3749 | 1.7914 | 2881 | 1.4506 |
| 1.4116 | 1.8331 | 2948 | 1.4468 |
| 1.3988 | 1.8747 | 3015 | 1.4456 |
| 1.2534 | 1.9164 | 3082 | 1.4448 |
| 1.3564 | 1.9580 | 3149 | 1.4412 |
| 1.3668 | 1.9997 | 3216 | 1.4392 |
### Framework versions
- Transformers 4.51.3
- Pytorch 2.4.1+cu121
- Datasets 3.5.1
- Tokenizers 0.21.1