Hugging Face's logo

llama-duo
/
gemma2b-summarize-gemini1_5flash-128k

PEFT
TensorBoard
Safetensors
gemma
alignment-handbook
trl
sft
Generated from Trainer
4-bit precision
bitsandbytes
Model card Files
xet
Metrics Community
chansung commited on
Commit
7ef99e8
·
verified ·
1 Parent(s): 092d8d3

Model save

Browse files
Files changed (6) hide show
  1. README.md +19 -25
  2. adapter_model.safetensors +1 -1
  3. all_results.json +6 -11
  4. runs/Jun10_07-38-05_48ddfe8e991f/events.out.tfevents.1718005108.48ddfe8e991f.245392.0 +2 -2
  5. train_results.json +6 -6
  6. trainer_state.json +0 -0
README.md CHANGED
@@ -2,13 +2,12 @@
2
  license: gemma
3
  library_name: peft
4
  tags:
5
- - alignment-handbook
6
  - trl
7
  - sft
8
  - generated_from_trainer
9
  base_model: google/gemma-2b
10
  datasets:
11
- - llama-duo/synth_summarize_dataset_dedup
12
  model-index:
13
  - name: gemma2b-summarize-gemini1_5flash-128k
14
  results: []
@@ -19,9 +18,9 @@ should probably proofread and complete it, then remove this comment. -->
19
 
20
  # gemma2b-summarize-gemini1_5flash-128k
21
 
22
- This model is a fine-tuned version of [google/gemma-2b](https://huggingface.co/google/gemma-2b) on the llama-duo/synth_summarize_dataset_dedup dataset.
23
  It achieves the following results on the evaluation set:
24
- - Loss: 2.5119
25
 
26
  ## Model description
27
 
@@ -45,40 +44,35 @@ The following hyperparameters were used during training:
45
  - eval_batch_size: 8
46
  - seed: 42
47
  - distributed_type: multi-GPU
48
- - num_devices: 4
49
  - gradient_accumulation_steps: 2
50
- - total_train_batch_size: 64
51
- - total_eval_batch_size: 32
52
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
53
  - lr_scheduler_type: cosine
54
  - lr_scheduler_warmup_ratio: 0.1
55
- - num_epochs: 15
56
 
57
  ### Training results
58
 
59
  | Training Loss | Epoch | Step | Validation Loss |
60
  |:-------------:|:-----:|:----:|:---------------:|
61
- | 1.1289 | 1.0 | 208 | 2.5162 |
62
- | 1.0298 | 2.0 | 416 | 2.4574 |
63
- | 0.9905 | 3.0 | 624 | 2.4455 |
64
- | 0.9668 | 4.0 | 832 | 2.4518 |
65
- | 0.9507 | 5.0 | 1040 | 2.4578 |
66
- | 0.9348 | 6.0 | 1248 | 2.4685 |
67
- | 0.9236 | 7.0 | 1456 | 2.4789 |
68
- | 0.9156 | 8.0 | 1664 | 2.4831 |
69
- | 0.8987 | 9.0 | 1872 | 2.4963 |
70
- | 0.9008 | 10.0 | 2080 | 2.5021 |
71
- | 0.8976 | 11.0 | 2288 | 2.5050 |
72
- | 0.8941 | 12.0 | 2496 | 2.5107 |
73
- | 0.8878 | 13.0 | 2704 | 2.5123 |
74
- | 0.8896 | 14.0 | 2912 | 2.5120 |
75
- | 0.8797 | 15.0 | 3120 | 2.5119 |
76
 
77
 
78
  ### Framework versions
79
 
80
  - PEFT 0.11.1
81
- - Transformers 4.40.1
82
- - Pytorch 2.2.0+cu121
83
  - Datasets 2.19.2
84
  - Tokenizers 0.19.1
 
2
  license: gemma
3
  library_name: peft
4
  tags:
 
5
  - trl
6
  - sft
7
  - generated_from_trainer
8
  base_model: google/gemma-2b
9
  datasets:
10
+ - generator
11
  model-index:
12
  - name: gemma2b-summarize-gemini1_5flash-128k
13
  results: []
 
18
 
19
  # gemma2b-summarize-gemini1_5flash-128k
20
 
21
+ This model is a fine-tuned version of [google/gemma-2b](https://huggingface.co/google/gemma-2b) on the generator dataset.
22
  It achieves the following results on the evaluation set:
23
+ - Loss: 2.5573
24
 
25
  ## Model description
26
 
 
44
  - eval_batch_size: 8
45
  - seed: 42
46
  - distributed_type: multi-GPU
47
+ - num_devices: 8
48
  - gradient_accumulation_steps: 2
49
+ - total_train_batch_size: 128
50
+ - total_eval_batch_size: 64
51
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
52
  - lr_scheduler_type: cosine
53
  - lr_scheduler_warmup_ratio: 0.1
54
+ - num_epochs: 10
55
 
56
  ### Training results
57
 
58
  | Training Loss | Epoch | Step | Validation Loss |
59
  |:-------------:|:-----:|:----:|:---------------:|
60
+ | 1.0978 | 1.0 | 104 | 2.4831 |
61
+ | 0.9985 | 2.0 | 208 | 2.4666 |
62
+ | 0.9543 | 3.0 | 312 | 2.4561 |
63
+ | 0.92 | 4.0 | 416 | 2.4799 |
64
+ | 0.9016 | 5.0 | 520 | 2.4990 |
65
+ | 0.8871 | 6.0 | 624 | 2.5250 |
66
+ | 0.8635 | 7.0 | 728 | 2.5363 |
67
+ | 0.8535 | 8.0 | 832 | 2.5546 |
68
+ | 0.845 | 9.0 | 936 | 2.5566 |
69
+ | 0.853 | 10.0 | 1040 | 2.5573 |
 
 
 
 
 
70
 
71
 
72
  ### Framework versions
73
 
74
  - PEFT 0.11.1
75
+ - Transformers 4.41.2
76
+ - Pytorch 2.3.1+cu121
77
  - Datasets 2.19.2
78
  - Tokenizers 0.19.1
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:4814f7f66589aaadfed01dc32a18d8571f439227757d94fe2ae89ec07c1876de
3
  size 78480320
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9a18143e465d21dcbf89d18140313cb39c82bc6b7dabfc909f3bac4791dbce41
3
  size 78480320
all_results.json CHANGED
@@ -1,14 +1,9 @@
1
  {
2
- "epoch": 15.0,
3
- "eval_loss": 2.511939525604248,
4
- "eval_runtime": 0.5077,
5
- "eval_samples": 25,
6
- "eval_samples_per_second": 19.697,
7
- "eval_steps_per_second": 1.97,
8
- "total_flos": 2.443473144564941e+18,
9
- "train_loss": 0.9840393823690904,
10
- "train_runtime": 10705.8296,
11
  "train_samples": 126706,
12
- "train_samples_per_second": 18.636,
13
- "train_steps_per_second": 0.291
14
  }
 
1
  {
2
+ "epoch": 10.0,
3
+ "total_flos": 1.6530423313701274e+18,
4
+ "train_loss": 0.977329820394516,
5
+ "train_runtime": 7126.3412,
 
 
 
 
 
6
  "train_samples": 126706,
7
+ "train_samples_per_second": 18.665,
8
+ "train_steps_per_second": 0.146
9
  }
runs/Jun10_07-38-05_48ddfe8e991f/events.out.tfevents.1718005108.48ddfe8e991f.245392.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:45270c1444049457bf3bf1086fc1c2bb641972d25d6781cd50553d6f477cf7a1
3
- size 50349
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1fd5987d98a2ba0a61e0d5e24e4a2f95a1c2c2baefe2e81f297e0f8aa8ab8684
3
+ size 52662
train_results.json CHANGED
@@ -1,9 +1,9 @@
1
  {
2
- "epoch": 15.0,
3
- "total_flos": 2.443473144564941e+18,
4
- "train_loss": 0.9840393823690904,
5
- "train_runtime": 10705.8296,
6
  "train_samples": 126706,
7
- "train_samples_per_second": 18.636,
8
- "train_steps_per_second": 0.291
9
  }
 
1
  {
2
+ "epoch": 10.0,
3
+ "total_flos": 1.6530423313701274e+18,
4
+ "train_loss": 0.977329820394516,
5
+ "train_runtime": 7126.3412,
6
  "train_samples": 126706,
7
+ "train_samples_per_second": 18.665,
8
+ "train_steps_per_second": 0.146
9
  }
trainer_state.json CHANGED
The diff for this file is too large to render. See raw diff