Model save

README.md

@@ -0,0 +1,236 @@
+---
+library_name: transformers
+license: apache-2.0
+base_model: Dans-DiscountModels/Mistral-Nemo-Base-2407-DanChat
+tags:
+- axolotl
+- generated_from_trainer
+datasets:
+- Dans-DiscountModels/pretokenization-test-5
+model-index:
+- name: 12b-mn-dans-personality-engine-v1.3.0-TestArticle-1
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
+<details><summary>See axolotl config</summary>
+
+axolotl version: `0.10.0.dev0`
+```yaml
+base_model: Dans-DiscountModels/Mistral-Nemo-Base-2407-DanChat
+model_type: AutoModelForCausalLM
+tokenizer_type: AutoTokenizer
+
+trust_remote_code:
+
+# wandb configuration
+wandb_project: 12b-mn-dans-personality-engine
+wandb_watch:
+
+wandb_run_id: V1.3.0-1-4 # V{Version}-{Run Number}-{Attempt Number}
+wandb_log_model:
+
+# push checkpoints to hub
+hub_model_id: Dans-DiscountModels/12b-mn-dans-personality-engine-v1.3.0-TestArticle-1
+# how to push checkpoints to hub
+# https://huggingface.co/docs/transformers/v4.31.0/en/main_classes/trainer#transformers.TrainingArguments.hub_strategy
+hub_strategy: "every_save"
+# Whether to use hf `use_auth_token` for loading datasets. Useful for fetching private datasets
+# Required to be true when used in combination with `push_dataset_to_hub`
+hf_use_auth_token: true
+
+# where to save the finished model to
+output_dir: ./12b-mn-dans-personality-engine-v1.3.0
+
+# dataset settings (local or huggingface repo)
+datasets:
+  - path: Dans-DiscountModels/pretokenization-test-5
+    ds_type: parquet
+    type:
+
+plugins:
+  - axolotl.integrations.liger.LigerPlugin
+  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
+liger_rope: true
+liger_rms_norm: true
+liger_layer_norm: true
+liger_glu_activation: true
+liger_fused_linear_cross_entropy: true
+cut_cross_entropy: true
+
+load_in_8bit: false
+load_in_4bit: false
+strict: false
+
+adapter:
+lora_model_dir:
+
+dataset_prepared_path: ./12b-mn-dans-personality-engine-data
+val_set_size: 0.003
+
+sequence_len: 32768
+
+sample_packing: true
+eval_sample_packing: true
+
+pad_to_sequence_len: true
+
+gradient_checkpointing: true
+
+gradient_accumulation_steps: 2
+micro_batch_size: 2
+
+num_epochs: 2
+
+optimizer: ademamix_8bit
+optim_args: "beta1=0.9,beta2=0.999,beta3=0.999,alpha=5"
+
+lr_scheduler: rex
+learning_rate: 0.00001
+cosine_min_lr_ratio:
+
+weight_decay:
+
+max_grad_norm: 0.001
+
+train_on_inputs: false
+group_by_length: false
+
+bf16: true
+fp16: false
+tf32: false
+
+early_stopping_patience:
+
+resume_from_checkpoint:
+auto_resume_from_checkpoints: true
+
+local_rank:
+logging_steps: 1
+xformers_attention:
+flash_attention: true
+
+warmup_ratio: 0.1
+
+evals_per_epoch: 24
+eval_table_size:
+eval_max_new_tokens:
+
+saves_per_epoch: 2
+save_total_limit: 1
+
+debug: false
+
+deepspeed: deepspeed_configs/zero3_bf16.json
+
+fsdp:
+fsdp_config:
+
+special_tokens:
+
+```
+
+</details><br>
+
+# 12b-mn-dans-personality-engine-v1.3.0-TestArticle-1
+
+This model is a fine-tuned version of [Dans-DiscountModels/Mistral-Nemo-Base-2407-DanChat](https://huggingface.co/Dans-DiscountModels/Mistral-Nemo-Base-2407-DanChat) on the Dans-DiscountModels/pretokenization-test-5 dataset.
+It achieves the following results on the evaluation set:
+- Loss: 1.4392
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 1e-05
+- train_batch_size: 2
+- eval_batch_size: 2
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 8
+- gradient_accumulation_steps: 2
+- total_train_batch_size: 32
+- total_eval_batch_size: 16
+- optimizer: Use ademamix_8bit and the args are:
+beta1=0.9,beta2=0.999,beta3=0.999,alpha=5
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 321
+- num_epochs: 2.0
+
+### Training results
+
+| Training Loss | Epoch  | Step | Validation Loss |
+|:-------------:|:------:|:----:|:---------------:|
+| 1.8086        | 0.0006 | 1    | 1.7459          |
+| 1.593         | 0.0417 | 67   | 1.5911          |
+| 1.5578        | 0.0833 | 134  | 1.5565          |
+| 1.5782        | 0.1250 | 201  | 1.5436          |
+| 1.5702        | 0.1666 | 268  | 1.5377          |
+| 1.5926        | 0.2083 | 335  | 1.5328          |
+| 1.6364        | 0.2499 | 402  | 1.5291          |
+| 1.5082        | 0.2916 | 469  | 1.5234          |
+| 1.6002        | 0.3332 | 536  | 1.5197          |
+| 1.5252        | 0.3749 | 603  | 1.5162          |
+| 1.5915        | 0.4165 | 670  | 1.5121          |
+| 1.5108        | 0.4582 | 737  | 1.5103          |
+| 1.5663        | 0.4998 | 804  | 1.5063          |
+| 1.5085        | 0.5415 | 871  | 1.5037          |
+| 1.4273        | 0.5832 | 938  | 1.5024          |
+| 1.5528        | 0.6248 | 1005 | 1.4994          |
+| 1.6072        | 0.6665 | 1072 | 1.4975          |
+| 1.6074        | 0.7081 | 1139 | 1.4920          |
+| 1.5495        | 0.7498 | 1206 | 1.4904          |
+| 1.6117        | 0.7914 | 1273 | 1.4883          |
+| 1.4621        | 0.8331 | 1340 | 1.4850          |
+| 1.6381        | 0.8747 | 1407 | 1.4838          |
+| 1.4221        | 0.9164 | 1474 | 1.4813          |
+| 1.5812        | 0.9580 | 1541 | 1.4789          |
+| 1.4581        | 0.9997 | 1608 | 1.4750          |
+| 1.4608        | 1.0417 | 1675 | 1.4800          |
+| 1.5261        | 1.0833 | 1742 | 1.4798          |
+| 1.3856        | 1.1250 | 1809 | 1.4796          |
+| 1.4469        | 1.1666 | 1876 | 1.4766          |
+| 1.4783        | 1.2083 | 1943 | 1.4741          |
+| 1.5025        | 1.2499 | 2010 | 1.4733          |
+| 1.4531        | 1.2916 | 2077 | 1.4726          |
+| 1.4719        | 1.3332 | 2144 | 1.4712          |
+| 1.4123        | 1.3749 | 2211 | 1.4700          |
+| 1.4653        | 1.4165 | 2278 | 1.4673          |
+| 1.4571        | 1.4582 | 2345 | 1.4660          |
+| 1.4261        | 1.4998 | 2412 | 1.4660          |
+| 1.3212        | 1.5415 | 2479 | 1.4620          |
+| 1.3828        | 1.5832 | 2546 | 1.4617          |
+| 1.3617        | 1.6248 | 2613 | 1.4597          |
+| 1.4364        | 1.6665 | 2680 | 1.4567          |
+| 1.4686        | 1.7081 | 2747 | 1.4549          |
+| 1.3317        | 1.7498 | 2814 | 1.4530          |
+| 1.3749        | 1.7914 | 2881 | 1.4506          |
+| 1.4116        | 1.8331 | 2948 | 1.4468          |
+| 1.3988        | 1.8747 | 3015 | 1.4456          |
+| 1.2534        | 1.9164 | 3082 | 1.4448          |
+| 1.3564        | 1.9580 | 3149 | 1.4412          |
+| 1.3668        | 1.9997 | 3216 | 1.4392          |
+
+
+### Framework versions
+
+- Transformers 4.51.3
+- Pytorch 2.4.1+cu121
+- Datasets 3.5.1
+- Tokenizers 0.21.1