SystemAdmin123
/

test-repo

@@ -1,6 +1,6 @@
 ---
-library_name: transformers
-base_model: trl-internal-testing/tiny-random-LlamaForCausalLM
 tags:
 - axolotl
 - generated_from_trainer
@@ -19,8 +19,9 @@ should probably proofread and complete it, then remove this comment. -->
 axolotl version: `0.6.0`
 ```yaml
-base_model: trl-internal-testing/tiny-random-LlamaForCausalLM
-batch_size: 128
 bf16: true
 chat_template: tokenizer_default_fallback_alpaca
 datasets:
@@ -36,36 +37,42 @@ datasets:
     system_prompt: ''
 device_map: auto
 eval_sample_packing: false
-eval_steps: 200
 flash_attention: true
 gradient_checkpointing: true
 group_by_length: true
 hub_model_id: SystemAdmin123/test-repo
 hub_strategy: checkpoint
-learning_rate: 0.0002
 logging_steps: 10
 lr_scheduler: cosine
-max_steps: 10000
-micro_batch_size: 32
 model_type: AutoModelForCausalLM
-num_epochs: 100
 optimizer: adamw_bnb_8bit
 output_dir: /root/.sn56/axolotl/tmp/test-repo
 pad_to_sequence_len: true
 resize_token_embeddings_to_32x: false
 sample_packing: true
-save_steps: 200
 save_total_limit: 1
 sequence_len: 2048
-tokenizer_type: LlamaTokenizerFast
 torch_dtype: bf16
 training_args_kwargs:
   hub_private_repo: true
 trust_remote_code: true
-val_set_size: 0.1
 wandb_entity: ''
 wandb_mode: online
-wandb_name: trl-internal-testing/tiny-random-LlamaForCausalLM-argilla/databricks-dolly-15k-curated-en
 wandb_project: Gradients-On-Demand
 wandb_run: your_name
 wandb_runid: default
@@ -77,7 +84,9 @@ warmup_ratio: 0.05
 # test-repo
-This model is a fine-tuned version of [trl-internal-testing/tiny-random-LlamaForCausalLM](https://huggingface.co/trl-internal-testing/tiny-random-LlamaForCausalLM) on the argilla/databricks-dolly-15k-curated-en dataset.
 ## Model description
@@ -96,29 +105,40 @@ More information needed
 ### Training hyperparameters
 The following hyperparameters were used during training:
-- learning_rate: 0.0002
-- train_batch_size: 32
-- eval_batch_size: 32
 - seed: 42
 - distributed_type: multi-GPU
 - num_devices: 4
-- total_train_batch_size: 128
-- total_eval_batch_size: 128
-- optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: cosine
-- lr_scheduler_warmup_steps: 5
-- training_steps: 100
 ### Training results
 | Training Loss | Epoch  | Step | Validation Loss |
 |:-------------:|:------:|:----:|:---------------:|
-| No log        | 0.1667 | 1    | 10.3764         |
 ### Framework versions
-- Transformers 4.48.1
-- Pytorch 2.5.1+cu124
-- Datasets 3.2.0
-- Tokenizers 0.21.0

 ---
+library_name: peft
+base_model: peft-internal-testing/tiny-dummy-qwen2
 tags:
 - axolotl
 - generated_from_trainer
 axolotl version: `0.6.0`
 ```yaml
+adapter: lora
+base_model: peft-internal-testing/tiny-dummy-qwen2
+batch_size: 64
 bf16: true
 chat_template: tokenizer_default_fallback_alpaca
 datasets:
     system_prompt: ''
 device_map: auto
 eval_sample_packing: false
+eval_steps: 0.1
 flash_attention: true
 gradient_checkpointing: true
 group_by_length: true
 hub_model_id: SystemAdmin123/test-repo
 hub_strategy: checkpoint
+learning_rate: 0.0001
 logging_steps: 10
+lora_alpha: 256
+lora_dropout: 0.1
+lora_r: 128
+lora_target_linear: true
 lr_scheduler: cosine
+max_steps: 160.0
+micro_batch_size: 7
 model_type: AutoModelForCausalLM
+num_epochs: 10000
 optimizer: adamw_bnb_8bit
 output_dir: /root/.sn56/axolotl/tmp/test-repo
 pad_to_sequence_len: true
 resize_token_embeddings_to_32x: false
 sample_packing: true
+save_steps: 40
 save_total_limit: 1
 sequence_len: 2048
+tokenizer_type: Qwen2TokenizerFast
 torch_dtype: bf16
 training_args_kwargs:
+  disable_tqdm: true
   hub_private_repo: true
+  save_only_model: true
 trust_remote_code: true
+val_set_size: 0.01
 wandb_entity: ''
 wandb_mode: online
+wandb_name: peft-internal-testing/tiny-dummy-qwen2-argilla/databricks-dolly-15k-curated-en
 wandb_project: Gradients-On-Demand
 wandb_run: your_name
 wandb_runid: default
 # test-repo
+This model is a fine-tuned version of [peft-internal-testing/tiny-dummy-qwen2](https://huggingface.co/peft-internal-testing/tiny-dummy-qwen2) on the argilla/databricks-dolly-15k-curated-en dataset.
+It achieves the following results on the evaluation set:
+- Loss: 11.9145
 ## Model description
 ### Training hyperparameters
 The following hyperparameters were used during training:
+- learning_rate: 0.0001
+- train_batch_size: 7
+- eval_batch_size: 7
 - seed: 42
 - distributed_type: multi-GPU
 - num_devices: 4
+- total_train_batch_size: 28
+- total_eval_batch_size: 28
+- optimizer: Use adamw_bnb_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 8
+- training_steps: 160
 ### Training results
 | Training Loss | Epoch  | Step | Validation Loss |
 |:-------------:|:------:|:----:|:---------------:|
+| No log        | 0.0345 | 1    | 11.9318         |
+| 11.9309       | 0.5517 | 16   | 11.9279         |
+| 11.9236       | 1.1034 | 32   | 11.9212         |
+| 11.9214       | 1.6552 | 48   | 11.9193         |
+| 11.9198       | 2.2069 | 64   | 11.9178         |
+| 11.9188       | 2.7586 | 80   | 11.9161         |
+| 11.9181       | 3.3103 | 96   | 11.9151         |
+| 11.9175       | 3.8621 | 112  | 11.9147         |
+| 11.9174       | 4.4138 | 128  | 11.9143         |
+| 11.917        | 4.9655 | 144  | 11.9140         |
+| 11.9169       | 5.5172 | 160  | 11.9145         |
 ### Framework versions
+- PEFT 0.14.0
+- Transformers 4.47.1
+- Pytorch 2.3.1+cu121
+- Datasets 3.1.0
+- Tokenizers 0.21.0

adapter_model.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:70c7833783fa3fc74e85a17679bf42d7fc78f79ffd5be3a976fc6ea108a186a6
+size 190338