Hugging Face's logo

BeaverAI
/
MS-2501-DPE-QwQify-v0.1-24B-LoRA-WS

PEFT
Safetensors
mistral
axolotl
Generated from Trainer
Model card Files Community
g4rg commited on
Commit
6b069b1
·
verified ·
1 Parent(s): f43ae3f

Model save

Browse files
Files changed (1) hide show
  1. README.md +290 -0
README.md ADDED
@@ -0,0 +1,290 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: peft
3
+ license: apache-2.0
4
+ base_model: PocketDoc/Dans-PersonalityEngine-V1.2.0-24b
5
+ tags:
6
+ - axolotl
7
+ - generated_from_trainer
8
+ datasets:
9
+ - PJMixers-Dev/allura-org_gryphe-sonnet-3.5-charcards-names-added-qwq-all-aphrodite-Shuffled
10
+ - PJMixers-Dev/anthracite-org_c2_logs_32k_llama3_qwen2_v1.3-qwq-all-aphrodite-Shuffled
11
+ - PJMixers-Dev/grimulkan_aicg-logs-augmented-system-qwq-all-aphrodite-Shuffled
12
+ - PJMixers-Dev/grimulkan_jannie-log-augmented-system-qwq-all-aphrodite-Shuffled
13
+ - PJMixers-Dev/grimulkan_PIPPA-augmented-dedup-system-qwq-all-aphrodite-Shuffled
14
+ - PJMixers-Dev/lemonilia_LimaRP-Only-NonSus-Simple-CustomShareGPT-qwq-all-aphrodite-Shuffled
15
+ - PJMixers-Dev/MinervaAI_Aesir-Preview-Anon-qwq-all-aphrodite-Shuffled
16
+ - PJMixers-Dev/NyxKrage_chub-logs-sharegpt-longest-CustomShareGPT-qwq-all-aphrodite-Shuffled
17
+ - PJMixers-Dev/PocketDoc_Dans-Prosemaxx-Cowriter-XL-8192-shrunk-l3-qwq-all-aphrodite-Shuffled
18
+ - PJMixers-Dev/PocketDoc_Dans-Personamaxx-Rainy-qwq-all-aphrodite-Shuffled
19
+ model-index:
20
+ - name: MS-2501-DPE-QwQify-v0.1-24B-LoRA-WS
21
+ results: []
22
+ ---
23
+
24
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
25
+ should probably proofread and complete it, then remove this comment. -->
26
+
27
+ [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
28
+ <details><summary>See axolotl config</summary>
29
+
30
+ axolotl version: `0.8.0.dev0`
31
+ ```yaml
32
+ mlflow_tracking_uri: http://127.0.0.1:7860
33
+ mlflow_experiment_name: MS-2501-DPE-QwQify-v0.1-24B-LoRA
34
+
35
+ # Hugging Face saving config
36
+ hub_model_id: BeaverAI/MS-2501-DPE-QwQify-v0.1-24B-LoRA-WS
37
+ hub_strategy: every_save
38
+
39
+ # Model checkpointing config
40
+ output_dir: ./Outputs/MS-2501-DPE-QwQify-v0.1-24B-LoRA
41
+ resume_from_checkpoint:
42
+ save_steps: 50
43
+ save_safetensors: true
44
+ save_total_limit: 3
45
+ save_only_model: false
46
+
47
+ # Model architecture config
48
+ base_model: PocketDoc/Dans-PersonalityEngine-V1.2.0-24b
49
+ model_type: MistralForCausalLM
50
+ tokenizer_type: AutoTokenizer
51
+
52
+ # Mixed precision training config
53
+ bf16: true
54
+ fp16: false
55
+ tf32: false
56
+
57
+ # Model loading config
58
+ load_in_8bit: false
59
+ load_in_4bit: false
60
+ strict: false
61
+
62
+ # Sequence config
63
+ sequence_len: 8192
64
+ min_sample_len: 256
65
+ sample_packing: true
66
+ eval_sample_packing: true
67
+ pad_to_sequence_len: true
68
+ train_on_inputs: false
69
+ group_by_length: false
70
+
71
+ # LoRA adapter config
72
+ adapter: lora
73
+ lora_model_dir:
74
+ lora_r: 128
75
+ lora_alpha: 128
76
+ lora_dropout: 0.125
77
+ peft_layers_to_transform:
78
+ peft_use_dora:
79
+ peft_use_rslora:
80
+ peft_layer_replication:
81
+ lora_target_modules:
82
+ - gate_proj
83
+ - down_proj
84
+ - up_proj
85
+ - q_proj
86
+ - v_proj
87
+ - k_proj
88
+ - o_proj
89
+ lora_modules_to_save:
90
+
91
+ # Fix uninitialized tokens (such as <|start_header_id|> on the base L3 models)
92
+ fix_untrained_tokens:
93
+
94
+ # Dataset config
95
+ # https://github.com/xzuyn/axolotl/blob/came-plus-formatters/src/axolotl/prompt_strategies/customchatml-regex-last-only.py
96
+ datasets:
97
+ - path: PJMixers-Dev/allura-org_gryphe-sonnet-3.5-charcards-names-added-qwq-all-aphrodite-Shuffled
98
+ split: train
99
+ type: customchatml-regex-last-only
100
+ - path: PJMixers-Dev/anthracite-org_c2_logs_32k_llama3_qwen2_v1.3-qwq-all-aphrodite-Shuffled
101
+ split: train
102
+ type: customchatml-regex-last-only
103
+ - path: PJMixers-Dev/grimulkan_aicg-logs-augmented-system-qwq-all-aphrodite-Shuffled
104
+ split: train
105
+ type: customchatml-regex-last-only
106
+ - path: PJMixers-Dev/grimulkan_jannie-log-augmented-system-qwq-all-aphrodite-Shuffled
107
+ split: train
108
+ type: customchatml-regex-last-only
109
+ - path: PJMixers-Dev/grimulkan_PIPPA-augmented-dedup-system-qwq-all-aphrodite-Shuffled
110
+ split: train
111
+ type: customchatml-regex-last-only
112
+ - path: PJMixers-Dev/lemonilia_LimaRP-Only-NonSus-Simple-CustomShareGPT-qwq-all-aphrodite-Shuffled
113
+ split: train
114
+ type: customchatml-regex-last-only
115
+ - path: PJMixers-Dev/MinervaAI_Aesir-Preview-Anon-qwq-all-aphrodite-Shuffled
116
+ split: train
117
+ type: customchatml-regex-last-only
118
+ - path: PJMixers-Dev/NyxKrage_chub-logs-sharegpt-longest-CustomShareGPT-qwq-all-aphrodite-Shuffled
119
+ split: train
120
+ type: customchatml-regex-last-only
121
+ - path: PJMixers-Dev/PocketDoc_Dans-Prosemaxx-Cowriter-XL-8192-shrunk-l3-qwq-all-aphrodite-Shuffled
122
+ split: train
123
+ type: customchatml-regex-last-only
124
+ - path: PJMixers-Dev/PocketDoc_Dans-Personamaxx-Rainy-qwq-all-aphrodite-Shuffled
125
+ split: train
126
+ type: customchatml-regex-last-only
127
+ test_datasets:
128
+ - path: PJMixers-Dev/allura-org_gryphe-sonnet-3.5-charcards-names-added-qwq-all-aphrodite-Shuffled
129
+ split: test
130
+ type: customchatml-regex-last-only
131
+ - path: PJMixers-Dev/anthracite-org_c2_logs_32k_llama3_qwen2_v1.3-qwq-all-aphrodite-Shuffled
132
+ split: test
133
+ type: customchatml-regex-last-only
134
+ - path: PJMixers-Dev/grimulkan_aicg-logs-augmented-system-qwq-all-aphrodite-Shuffled
135
+ split: test
136
+ type: customchatml-regex-last-only
137
+ - path: PJMixers-Dev/grimulkan_jannie-log-augmented-system-qwq-all-aphrodite-Shuffled
138
+ split: test
139
+ type: customchatml-regex-last-only
140
+ - path: PJMixers-Dev/grimulkan_PIPPA-augmented-dedup-system-qwq-all-aphrodite-Shuffled
141
+ split: test
142
+ type: customchatml-regex-last-only
143
+ - path: PJMixers-Dev/lemonilia_LimaRP-Only-NonSus-Simple-CustomShareGPT-qwq-all-aphrodite-Shuffled
144
+ split: test
145
+ type: customchatml-regex-last-only
146
+ - path: PJMixers-Dev/MinervaAI_Aesir-Preview-Anon-qwq-all-aphrodite-Shuffled
147
+ split: test
148
+ type: customchatml-regex-last-only
149
+ - path: PJMixers-Dev/NyxKrage_chub-logs-sharegpt-longest-CustomShareGPT-qwq-all-aphrodite-Shuffled
150
+ split: test
151
+ type: customchatml-regex-last-only
152
+ - path: PJMixers-Dev/PocketDoc_Dans-Prosemaxx-Cowriter-XL-8192-shrunk-l3-qwq-all-aphrodite-Shuffled
153
+ split: test
154
+ type: customchatml-regex-last-only
155
+ - path: PJMixers-Dev/PocketDoc_Dans-Personamaxx-Rainy-qwq-all-aphrodite-Shuffled
156
+ split: test
157
+ type: customchatml-regex-last-only
158
+ val_set_size: 0
159
+ eval_strategy: steps
160
+ eval_steps: 50
161
+ dataset_prepared_path: ./00-Tokenized-Datasets/MS-2501-DPE-QwQify-v0.1-24B-customchatml-regex-last-only
162
+ shuffle_merged_datasets: true
163
+ dataset_processes:
164
+
165
+ # Training hyperparameters
166
+ num_epochs: 2
167
+ gradient_accumulation_steps: 1
168
+ micro_batch_size: 8 # x4 GPUs = 32
169
+ eval_batch_size: 8 # x4 GPUs = 32
170
+ warmup_steps: 0
171
+ optimizer: came_pytorch
172
+ optim_args:
173
+ optim_target_modules:
174
+ lr_scheduler: rex
175
+ learning_rate: 2e-5
176
+ cosine_min_lr_ratio:
177
+ loraplus_lr_ratio:
178
+ loraplus_lr_embedding:
179
+ weight_decay: 0.1
180
+ max_grad_norm: 1
181
+ logging_steps: 1
182
+
183
+ # Model optimization
184
+ gradient_checkpointing: unsloth
185
+ flash_attention: true
186
+ plugins:
187
+ - axolotl.integrations.liger.LigerPlugin
188
+ - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
189
+ cut_cross_entropy: true
190
+ liger_rope: true
191
+ liger_rms_norm: true
192
+ liger_layer_norm: true
193
+ liger_glu_activation: true
194
+ liger_cross_entropy: false
195
+ liger_fused_linear_cross_entropy: false
196
+ lora_mlp_kernel: false
197
+ lora_qkv_kernel: false
198
+ lora_o_kernel: false
199
+
200
+ # DeepSpeed
201
+ deepspeed: deepspeed_configs/zero3_bf16.json
202
+
203
+ # Garbage Collection
204
+ gc_steps: 1
205
+
206
+ # Debug config
207
+ debug: true
208
+ seed: 42
209
+
210
+ # Token config
211
+ special_tokens:
212
+ bos_token: "<s>"
213
+ eos_token: "<|im_end|>"
214
+ pad_token: "<pad>"
215
+ tokens:
216
+
217
+ ```
218
+
219
+ </details><br>
220
+
221
+ # MS-2501-DPE-QwQify-v0.1-24B-LoRA-WS
222
+
223
+ This model is a fine-tuned version of [PocketDoc/Dans-PersonalityEngine-V1.2.0-24b](https://huggingface.co/PocketDoc/Dans-PersonalityEngine-V1.2.0-24b) on the PJMixers-Dev/allura-org_gryphe-sonnet-3.5-charcards-names-added-qwq-all-aphrodite-Shuffled, the PJMixers-Dev/anthracite-org_c2_logs_32k_llama3_qwen2_v1.3-qwq-all-aphrodite-Shuffled, the PJMixers-Dev/grimulkan_aicg-logs-augmented-system-qwq-all-aphrodite-Shuffled, the PJMixers-Dev/grimulkan_jannie-log-augmented-system-qwq-all-aphrodite-Shuffled, the PJMixers-Dev/grimulkan_PIPPA-augmented-dedup-system-qwq-all-aphrodite-Shuffled, the PJMixers-Dev/lemonilia_LimaRP-Only-NonSus-Simple-CustomShareGPT-qwq-all-aphrodite-Shuffled, the PJMixers-Dev/MinervaAI_Aesir-Preview-Anon-qwq-all-aphrodite-Shuffled, the PJMixers-Dev/NyxKrage_chub-logs-sharegpt-longest-CustomShareGPT-qwq-all-aphrodite-Shuffled, the PJMixers-Dev/PocketDoc_Dans-Prosemaxx-Cowriter-XL-8192-shrunk-l3-qwq-all-aphrodite-Shuffled and the PJMixers-Dev/PocketDoc_Dans-Personamaxx-Rainy-qwq-all-aphrodite-Shuffled datasets.
224
+ It achieves the following results on the evaluation set:
225
+ - Loss: 1.1949
226
+
227
+ ## Model description
228
+
229
+ More information needed
230
+
231
+ ## Intended uses & limitations
232
+
233
+ More information needed
234
+
235
+ ## Training and evaluation data
236
+
237
+ More information needed
238
+
239
+ ## Training procedure
240
+
241
+ ### Training hyperparameters
242
+
243
+ The following hyperparameters were used during training:
244
+ - learning_rate: 2e-05
245
+ - train_batch_size: 8
246
+ - eval_batch_size: 8
247
+ - seed: 42
248
+ - distributed_type: multi-GPU
249
+ - num_devices: 4
250
+ - total_train_batch_size: 32
251
+ - total_eval_batch_size: 32
252
+ - optimizer: Use OptimizerNames.ADAMW_HF with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
253
+ - lr_scheduler_type: cosine
254
+ - num_epochs: 2.0
255
+
256
+ ### Training results
257
+
258
+ | Training Loss | Epoch | Step | Validation Loss |
259
+ |:-------------:|:------:|:----:|:---------------:|
260
+ | 1.9925 | 0.0019 | 1 | 1.9225 |
261
+ | 1.4228 | 0.0936 | 50 | 1.4329 |
262
+ | 1.3473 | 0.1873 | 100 | 1.3722 |
263
+ | 1.3259 | 0.2809 | 150 | 1.3414 |
264
+ | 1.2795 | 0.3745 | 200 | 1.3199 |
265
+ | 1.2817 | 0.4682 | 250 | 1.3029 |
266
+ | 1.2365 | 0.5618 | 300 | 1.2910 |
267
+ | 1.2134 | 0.6554 | 350 | 1.2803 |
268
+ | 1.2655 | 0.7491 | 400 | 1.2700 |
269
+ | 1.2297 | 0.8427 | 450 | 1.2614 |
270
+ | 1.178 | 0.9363 | 500 | 1.2524 |
271
+ | 1.1525 | 1.0300 | 550 | 1.2467 |
272
+ | 1.1751 | 1.1236 | 600 | 1.2411 |
273
+ | 1.216 | 1.2172 | 650 | 1.2366 |
274
+ | 1.1706 | 1.3109 | 700 | 1.2302 |
275
+ | 1.1363 | 1.4045 | 750 | 1.2256 |
276
+ | 1.1563 | 1.4981 | 800 | 1.2194 |
277
+ | 1.1559 | 1.5918 | 850 | 1.2147 |
278
+ | 1.1263 | 1.6854 | 900 | 1.2090 |
279
+ | 1.099 | 1.7790 | 950 | 1.2038 |
280
+ | 1.1786 | 1.8727 | 1000 | 1.1994 |
281
+ | 1.1057 | 1.9663 | 1050 | 1.1949 |
282
+
283
+
284
+ ### Framework versions
285
+
286
+ - PEFT 0.14.0
287
+ - Transformers 4.49.0
288
+ - Pytorch 2.6.0+cu124
289
+ - Datasets 3.2.0
290
+ - Tokenizers 0.21.1