Hugging Face's logo

weege007
/
Qwen2.5-0.5B-Instruct_grpo_Countdown-Tasks-3to4

Text Generation
Transformers
TensorBoard
Safetensors
qwen2
Generated from Trainer
trl
grpo
conversational
text-generation-inference
Model card Files Metrics Community
weege007 commited on
Commit
69cd9b6
·
verified ·
1 Parent(s): b1b9809

Model save

Browse files
Files changed (6) hide show
  1. README.md +67 -0
  2. all_results.json +8 -0
  3. config.json +1 -1
  4. generation_config.json +14 -0
  5. train_results.json +8 -0
  6. trainer_state.json +1443 -0
README.md ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ model_name: Qwen2.5-0.5B-Instruct_grpo_Countdown-Tasks-3to4
4
+ tags:
5
+ - generated_from_trainer
6
+ - trl
7
+ - grpo
8
+ licence: license
9
+ ---
10
+
11
+ # Model Card for Qwen2.5-0.5B-Instruct_grpo_Countdown-Tasks-3to4
12
+
13
+ This model is a fine-tuned version of [None](https://huggingface.co/None).
14
+ It has been trained using [TRL](https://github.com/huggingface/trl).
15
+
16
+ ## Quick start
17
+
18
+ ```python
19
+ from transformers import pipeline
20
+
21
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
22
+ generator = pipeline("text-generation", model="weege007/Qwen2.5-0.5B-Instruct_grpo_Countdown-Tasks-3to4", device="cuda")
23
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
24
+ print(output["generated_text"])
25
+ ```
26
+
27
+ ## Training procedure
28
+
29
+ [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/weege007/huggingface/runs/qklw2es4)
30
+
31
+
32
+ This model was trained with GRPO, a method introduced in [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300).
33
+
34
+ ### Framework versions
35
+
36
+ - TRL: 0.18.0
37
+ - Transformers: 4.52.4
38
+ - Pytorch: 2.6.0+cu124
39
+ - Datasets: 3.3.2
40
+ - Tokenizers: 0.21.1
41
+
42
+ ## Citations
43
+
44
+ Cite GRPO as:
45
+
46
+ ```bibtex
47
+ @article{zhihong2024deepseekmath,
48
+ title = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},
49
+ author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},
50
+ year = 2024,
51
+ eprint = {arXiv:2402.03300},
52
+ }
53
+
54
+ ```
55
+
56
+ Cite TRL as:
57
+
58
+ ```bibtex
59
+ @misc{vonwerra2022trl,
60
+ title = {{TRL: Transformer Reinforcement Learning}},
61
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
62
+ year = 2020,
63
+ journal = {GitHub repository},
64
+ publisher = {GitHub},
65
+ howpublished = {\url{https://github.com/huggingface/trl}}
66
+ }
67
+ ```
all_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "total_flos": 0.0,
3
+ "train_loss": 8.04626297735922e-06,
4
+ "train_runtime": 2640.6632,
5
+ "train_samples": 441327,
6
+ "train_samples_per_second": 0.303,
7
+ "train_steps_per_second": 0.038
8
+ }
config.json CHANGED
@@ -22,7 +22,7 @@
22
  "tie_word_embeddings": true,
23
  "torch_dtype": "float32",
24
  "transformers_version": "4.52.4",
25
- "use_cache": false,
26
  "use_sliding_window": false,
27
  "vocab_size": 151936
28
  }
 
22
  "tie_word_embeddings": true,
23
  "torch_dtype": "float32",
24
  "transformers_version": "4.52.4",
25
+ "use_cache": true,
26
  "use_sliding_window": false,
27
  "vocab_size": 151936
28
  }
generation_config.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 151643,
3
+ "do_sample": true,
4
+ "eos_token_id": [
5
+ 151645,
6
+ 151643
7
+ ],
8
+ "pad_token_id": 151643,
9
+ "repetition_penalty": 1.1,
10
+ "temperature": 0.7,
11
+ "top_k": 20,
12
+ "top_p": 0.8,
13
+ "transformers_version": "4.52.4"
14
+ }
train_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "total_flos": 0.0,
3
+ "train_loss": 8.04626297735922e-06,
4
+ "train_runtime": 2640.6632,
5
+ "train_samples": 441327,
6
+ "train_samples_per_second": 0.303,
7
+ "train_steps_per_second": 0.038
8
+ }
trainer_state.json ADDED
@@ -0,0 +1,1443 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 0.00022658935437895256,
6
+ "eval_steps": 500,
7
+ "global_step": 100,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "clip_ratio/high_max": 0.0,
14
+ "clip_ratio/high_mean": 0.0,
15
+ "clip_ratio/low_mean": 0.0,
16
+ "clip_ratio/low_min": 0.0,
17
+ "clip_ratio/region_mean": 0.0,
18
+ "completions/clipped_ratio": 0.0625,
19
+ "completions/max_length": 656.0,
20
+ "completions/max_terminated_length": 222.5,
21
+ "completions/mean_length": 194.0625,
22
+ "completions/mean_terminated_length": 136.94643020629883,
23
+ "completions/min_length": 61.5,
24
+ "completions/min_terminated_length": 61.5,
25
+ "epoch": 4.531787087579051e-06,
26
+ "frac_reward_zero_std": 0.5,
27
+ "grad_norm": 12.525100708007812,
28
+ "kl": 0.0,
29
+ "learning_rate": 1.6666666666666665e-07,
30
+ "loss": -0.0,
31
+ "num_tokens": 5377.0,
32
+ "reward": 0.0625,
33
+ "reward_std": 0.1767766922712326,
34
+ "rewards/equation_reward_func/mean": 0.0,
35
+ "rewards/equation_reward_func/std": 0.0,
36
+ "rewards/format_reward_func/mean": 0.0625,
37
+ "rewards/format_reward_func/std": 0.1767766922712326,
38
+ "step": 2
39
+ },
40
+ {
41
+ "clip_ratio/high_max": 0.0,
42
+ "clip_ratio/high_mean": 0.0,
43
+ "clip_ratio/low_mean": 0.0,
44
+ "clip_ratio/low_min": 0.0,
45
+ "clip_ratio/region_mean": 0.0,
46
+ "completions/clipped_ratio": 0.0625,
47
+ "completions/max_length": 618.0,
48
+ "completions/max_terminated_length": 198.0,
49
+ "completions/mean_length": 171.25,
50
+ "completions/mean_terminated_length": 114.93750381469727,
51
+ "completions/min_length": 57.5,
52
+ "completions/min_terminated_length": 57.5,
53
+ "epoch": 9.063574175158102e-06,
54
+ "frac_reward_zero_std": 0.5,
55
+ "grad_norm": 0.001167318900115788,
56
+ "kl": 0.0009191570134134963,
57
+ "learning_rate": 5e-07,
58
+ "loss": 0.0,
59
+ "num_tokens": 10397.0,
60
+ "reward": 0.0625,
61
+ "reward_std": 0.1767766922712326,
62
+ "rewards/equation_reward_func/mean": 0.0,
63
+ "rewards/equation_reward_func/std": 0.0,
64
+ "rewards/format_reward_func/mean": 0.0625,
65
+ "rewards/format_reward_func/std": 0.1767766922712326,
66
+ "step": 4
67
+ },
68
+ {
69
+ "clip_ratio/high_max": 0.0,
70
+ "clip_ratio/high_mean": 0.0,
71
+ "clip_ratio/low_mean": 0.0,
72
+ "clip_ratio/low_min": 0.0,
73
+ "clip_ratio/region_mean": 0.0,
74
+ "completions/clipped_ratio": 0.0,
75
+ "completions/max_length": 477.0,
76
+ "completions/max_terminated_length": 477.0,
77
+ "completions/mean_length": 175.5,
78
+ "completions/mean_terminated_length": 175.5,
79
+ "completions/min_length": 38.5,
80
+ "completions/min_terminated_length": 38.5,
81
+ "epoch": 1.3595361262737154e-05,
82
+ "frac_reward_zero_std": 1.0,
83
+ "grad_norm": 0.0017322486964985728,
84
+ "kl": 0.0009386140663991682,
85
+ "learning_rate": 4.994757065594279e-07,
86
+ "loss": 0.0,
87
+ "num_tokens": 15493.0,
88
+ "reward": 0.0,
89
+ "reward_std": 0.0,
90
+ "rewards/equation_reward_func/mean": 0.0,
91
+ "rewards/equation_reward_func/std": 0.0,
92
+ "rewards/format_reward_func/mean": 0.0,
93
+ "rewards/format_reward_func/std": 0.0,
94
+ "step": 6
95
+ },
96
+ {
97
+ "clip_ratio/high_max": 0.0,
98
+ "clip_ratio/high_mean": 0.0,
99
+ "clip_ratio/low_mean": 0.0,
100
+ "clip_ratio/low_min": 0.0,
101
+ "clip_ratio/region_mean": 0.0,
102
+ "completions/clipped_ratio": 0.0,
103
+ "completions/max_length": 307.0,
104
+ "completions/max_terminated_length": 307.0,
105
+ "completions/mean_length": 156.25,
106
+ "completions/mean_terminated_length": 156.25,
107
+ "completions/min_length": 47.0,
108
+ "completions/min_terminated_length": 47.0,
109
+ "epoch": 1.8127148350316204e-05,
110
+ "frac_reward_zero_std": 0.5,
111
+ "grad_norm": 11.28031063079834,
112
+ "kl": 0.0007646345866305637,
113
+ "learning_rate": 4.979050253066063e-07,
114
+ "loss": 0.0,
115
+ "num_tokens": 20313.0,
116
+ "reward": 0.0625,
117
+ "reward_std": 0.1767766922712326,
118
+ "rewards/equation_reward_func/mean": 0.0,
119
+ "rewards/equation_reward_func/std": 0.0,
120
+ "rewards/format_reward_func/mean": 0.0625,
121
+ "rewards/format_reward_func/std": 0.1767766922712326,
122
+ "step": 8
123
+ },
124
+ {
125
+ "clip_ratio/high_max": 0.0,
126
+ "clip_ratio/high_mean": 0.0,
127
+ "clip_ratio/low_mean": 0.0,
128
+ "clip_ratio/low_min": 0.0,
129
+ "clip_ratio/region_mean": 0.0,
130
+ "completions/clipped_ratio": 0.0,
131
+ "completions/max_length": 312.5,
132
+ "completions/max_terminated_length": 312.5,
133
+ "completions/mean_length": 152.25,
134
+ "completions/mean_terminated_length": 152.25,
135
+ "completions/min_length": 65.0,
136
+ "completions/min_terminated_length": 65.0,
137
+ "epoch": 2.2658935437895258e-05,
138
+ "frac_reward_zero_std": 0.5,
139
+ "grad_norm": 9.902769088745117,
140
+ "kl": 0.0008557607743568951,
141
+ "learning_rate": 4.952945442245597e-07,
142
+ "loss": 0.0,
143
+ "num_tokens": 25061.0,
144
+ "reward": 0.0625,
145
+ "reward_std": 0.1767766922712326,
146
+ "rewards/equation_reward_func/mean": 0.0,
147
+ "rewards/equation_reward_func/std": 0.0,
148
+ "rewards/format_reward_func/mean": 0.0625,
149
+ "rewards/format_reward_func/std": 0.1767766922712326,
150
+ "step": 10
151
+ },
152
+ {
153
+ "clip_ratio/high_max": 0.0,
154
+ "clip_ratio/high_mean": 0.0,
155
+ "clip_ratio/low_mean": 0.0,
156
+ "clip_ratio/low_min": 0.0,
157
+ "clip_ratio/region_mean": 0.0,
158
+ "completions/clipped_ratio": 0.0,
159
+ "completions/max_length": 543.5,
160
+ "completions/max_terminated_length": 543.5,
161
+ "completions/mean_length": 164.4375,
162
+ "completions/mean_terminated_length": 164.4375,
163
+ "completions/min_length": 38.0,
164
+ "completions/min_terminated_length": 38.0,
165
+ "epoch": 2.7190722525474308e-05,
166
+ "frac_reward_zero_std": 0.5,
167
+ "grad_norm": 0.0027156081050634384,
168
+ "kl": 0.0010605865281831939,
169
+ "learning_rate": 4.916552125781528e-07,
170
+ "loss": 0.0,
171
+ "num_tokens": 29980.0,
172
+ "reward": 0.0625,
173
+ "reward_std": 0.1767766922712326,
174
+ "rewards/equation_reward_func/mean": 0.0,
175
+ "rewards/equation_reward_func/std": 0.0,
176
+ "rewards/format_reward_func/mean": 0.0625,
177
+ "rewards/format_reward_func/std": 0.1767766922712326,
178
+ "step": 12
179
+ },
180
+ {
181
+ "clip_ratio/high_max": 0.0,
182
+ "clip_ratio/high_mean": 0.0,
183
+ "clip_ratio/low_mean": 0.0,
184
+ "clip_ratio/low_min": 0.0,
185
+ "clip_ratio/region_mean": 0.0,
186
+ "completions/clipped_ratio": 0.0,
187
+ "completions/max_length": 451.0,
188
+ "completions/max_terminated_length": 451.0,
189
+ "completions/mean_length": 211.0,
190
+ "completions/mean_terminated_length": 211.0,
191
+ "completions/min_length": 70.5,
192
+ "completions/min_terminated_length": 70.5,
193
+ "epoch": 3.172250961305336e-05,
194
+ "frac_reward_zero_std": 0.5,
195
+ "grad_norm": 12.591176986694336,
196
+ "kl": 0.0009377936348755611,
197
+ "learning_rate": 4.870022949890676e-07,
198
+ "loss": 0.0,
199
+ "num_tokens": 35676.0,
200
+ "reward": 0.0625,
201
+ "reward_std": 0.1767766922712326,
202
+ "rewards/equation_reward_func/mean": 0.0,
203
+ "rewards/equation_reward_func/std": 0.0,
204
+ "rewards/format_reward_func/mean": 0.0625,
205
+ "rewards/format_reward_func/std": 0.1767766922712326,
206
+ "step": 14
207
+ },
208
+ {
209
+ "clip_ratio/high_max": 0.0,
210
+ "clip_ratio/high_mean": 0.0,
211
+ "clip_ratio/low_mean": 0.0,
212
+ "clip_ratio/low_min": 0.0,
213
+ "clip_ratio/region_mean": 0.0,
214
+ "completions/clipped_ratio": 0.0,
215
+ "completions/max_length": 465.5,
216
+ "completions/max_terminated_length": 465.5,
217
+ "completions/mean_length": 163.375,
218
+ "completions/mean_terminated_length": 163.375,
219
+ "completions/min_length": 49.0,
220
+ "completions/min_terminated_length": 49.0,
221
+ "epoch": 3.625429670063241e-05,
222
+ "frac_reward_zero_std": 0.0,
223
+ "grad_norm": 10.893583297729492,
224
+ "kl": 0.001199566273498931,
225
+ "learning_rate": 4.81355307410676e-07,
226
+ "loss": 0.0,
227
+ "num_tokens": 40570.0,
228
+ "reward": 0.125,
229
+ "reward_std": 0.3535533845424652,
230
+ "rewards/equation_reward_func/mean": 0.0,
231
+ "rewards/equation_reward_func/std": 0.0,
232
+ "rewards/format_reward_func/mean": 0.125,
233
+ "rewards/format_reward_func/std": 0.3535533845424652,
234
+ "step": 16
235
+ },
236
+ {
237
+ "clip_ratio/high_max": 0.0,
238
+ "clip_ratio/high_mean": 0.0,
239
+ "clip_ratio/low_mean": 0.0,
240
+ "clip_ratio/low_min": 0.0,
241
+ "clip_ratio/region_mean": 0.0,
242
+ "completions/clipped_ratio": 0.0,
243
+ "completions/max_length": 401.0,
244
+ "completions/max_terminated_length": 401.0,
245
+ "completions/mean_length": 209.75,
246
+ "completions/mean_terminated_length": 209.75,
247
+ "completions/min_length": 63.0,
248
+ "completions/min_terminated_length": 63.0,
249
+ "epoch": 4.078608378821146e-05,
250
+ "frac_reward_zero_std": 1.0,
251
+ "grad_norm": 0.0009183982037939131,
252
+ "kl": 0.0008469254862575326,
253
+ "learning_rate": 4.747379352713488e-07,
254
+ "loss": 0.0,
255
+ "num_tokens": 46174.0,
256
+ "reward": 0.0,
257
+ "reward_std": 0.0,
258
+ "rewards/equation_reward_func/mean": 0.0,
259
+ "rewards/equation_reward_func/std": 0.0,
260
+ "rewards/format_reward_func/mean": 0.0,
261
+ "rewards/format_reward_func/std": 0.0,
262
+ "step": 18
263
+ },
264
+ {
265
+ "clip_ratio/high_max": 0.0,
266
+ "clip_ratio/high_mean": 0.0,
267
+ "clip_ratio/low_mean": 0.0,
268
+ "clip_ratio/low_min": 0.0,
269
+ "clip_ratio/region_mean": 0.0,
270
+ "completions/clipped_ratio": 0.0,
271
+ "completions/max_length": 583.0,
272
+ "completions/max_terminated_length": 583.0,
273
+ "completions/mean_length": 211.125,
274
+ "completions/mean_terminated_length": 211.125,
275
+ "completions/min_length": 69.0,
276
+ "completions/min_terminated_length": 69.0,
277
+ "epoch": 4.5317870875790515e-05,
278
+ "frac_reward_zero_std": 1.0,
279
+ "grad_norm": 0.0012323512928560376,
280
+ "kl": 0.0012486951891332865,
281
+ "learning_rate": 4.6717793412953776e-07,
282
+ "loss": 0.0,
283
+ "num_tokens": 51832.0,
284
+ "reward": 0.0,
285
+ "reward_std": 0.0,
286
+ "rewards/equation_reward_func/mean": 0.0,
287
+ "rewards/equation_reward_func/std": 0.0,
288
+ "rewards/format_reward_func/mean": 0.0,
289
+ "rewards/format_reward_func/std": 0.0,
290
+ "step": 20
291
+ },
292
+ {
293
+ "clip_ratio/high_max": 0.0,
294
+ "clip_ratio/high_mean": 0.0,
295
+ "clip_ratio/low_mean": 0.0,
296
+ "clip_ratio/low_min": 0.0,
297
+ "clip_ratio/region_mean": 0.0,
298
+ "completions/clipped_ratio": 0.0625,
299
+ "completions/max_length": 620.5,
300
+ "completions/max_terminated_length": 285.5,
301
+ "completions/mean_length": 202.9375,
302
+ "completions/mean_terminated_length": 150.90178680419922,
303
+ "completions/min_length": 60.5,
304
+ "completions/min_terminated_length": 60.5,
305
+ "epoch": 4.984965796336956e-05,
306
+ "frac_reward_zero_std": 1.0,
307
+ "grad_norm": 0.0016945754177868366,
308
+ "kl": 0.00123220352907083,
309
+ "learning_rate": 4.5870701325731773e-07,
310
+ "loss": 0.0,
311
+ "num_tokens": 57327.0,
312
+ "reward": 0.0,
313
+ "reward_std": 0.0,
314
+ "rewards/equation_reward_func/mean": 0.0,
315
+ "rewards/equation_reward_func/std": 0.0,
316
+ "rewards/format_reward_func/mean": 0.0,
317
+ "rewards/format_reward_func/std": 0.0,
318
+ "step": 22
319
+ },
320
+ {
321
+ "clip_ratio/high_max": 0.0,
322
+ "clip_ratio/high_mean": 0.0,
323
+ "clip_ratio/low_mean": 0.0,
324
+ "clip_ratio/low_min": 0.0,
325
+ "clip_ratio/region_mean": 0.0,
326
+ "completions/clipped_ratio": 0.0625,
327
+ "completions/max_length": 683.0,
328
+ "completions/max_terminated_length": 434.5,
329
+ "completions/mean_length": 257.0,
330
+ "completions/mean_terminated_length": 209.44644165039062,
331
+ "completions/min_length": 84.5,
332
+ "completions/min_terminated_length": 84.5,
333
+ "epoch": 5.4381445050948616e-05,
334
+ "frac_reward_zero_std": 1.0,
335
+ "grad_norm": 0.00123355642426759,
336
+ "kl": 0.0011914248134416994,
337
+ "learning_rate": 4.4936070264068016e-07,
338
+ "loss": 0.0,
339
+ "num_tokens": 63719.0,
340
+ "reward": 0.0,
341
+ "reward_std": 0.0,
342
+ "rewards/equation_reward_func/mean": 0.0,
343
+ "rewards/equation_reward_func/std": 0.0,
344
+ "rewards/format_reward_func/mean": 0.0,
345
+ "rewards/format_reward_func/std": 0.0,
346
+ "step": 24
347
+ },
348
+ {
349
+ "clip_ratio/high_max": 0.0,
350
+ "clip_ratio/high_mean": 0.0,
351
+ "clip_ratio/low_mean": 0.0,
352
+ "clip_ratio/low_min": 0.0,
353
+ "clip_ratio/region_mean": 0.0,
354
+ "completions/clipped_ratio": 0.0625,
355
+ "completions/max_length": 641.0,
356
+ "completions/max_terminated_length": 235.5,
357
+ "completions/mean_length": 181.3125,
358
+ "completions/mean_terminated_length": 125.05357360839844,
359
+ "completions/min_length": 35.0,
360
+ "completions/min_terminated_length": 35.0,
361
+ "epoch": 5.891323213852767e-05,
362
+ "frac_reward_zero_std": 0.0,
363
+ "grad_norm": 13.299399375915527,
364
+ "kl": 0.0018970294222526718,
365
+ "learning_rate": 4.391782039544238e-07,
366
+ "loss": 0.0,
367
+ "num_tokens": 68924.0,
368
+ "reward": 0.125,
369
+ "reward_std": 0.3535533845424652,
370
+ "rewards/equation_reward_func/mean": 0.0,
371
+ "rewards/equation_reward_func/std": 0.0,
372
+ "rewards/format_reward_func/mean": 0.125,
373
+ "rewards/format_reward_func/std": 0.3535533845424652,
374
+ "step": 26
375
+ },
376
+ {
377
+ "clip_ratio/high_max": 0.0,
378
+ "clip_ratio/high_mean": 0.0,
379
+ "clip_ratio/low_mean": 0.0,
380
+ "clip_ratio/low_min": 0.0,
381
+ "clip_ratio/region_mean": 0.0,
382
+ "completions/clipped_ratio": 0.0,
383
+ "completions/max_length": 214.5,
384
+ "completions/max_terminated_length": 214.5,
385
+ "completions/mean_length": 108.6875,
386
+ "completions/mean_terminated_length": 108.6875,
387
+ "completions/min_length": 34.0,
388
+ "completions/min_terminated_length": 34.0,
389
+ "epoch": 6.344501922610672e-05,
390
+ "frac_reward_zero_std": 0.5,
391
+ "grad_norm": 0.0035000136122107506,
392
+ "kl": 0.0019113743601337774,
393
+ "learning_rate": 4.282022261367073e-07,
394
+ "loss": 0.0,
395
+ "num_tokens": 72919.0,
396
+ "reward": 0.0625,
397
+ "reward_std": 0.1767766922712326,
398
+ "rewards/equation_reward_func/mean": 0.0,
399
+ "rewards/equation_reward_func/std": 0.0,
400
+ "rewards/format_reward_func/mean": 0.0625,
401
+ "rewards/format_reward_func/std": 0.1767766922712326,
402
+ "step": 28
403
+ },
404
+ {
405
+ "clip_ratio/high_max": 0.0,
406
+ "clip_ratio/high_mean": 0.0,
407
+ "clip_ratio/low_mean": 0.0,
408
+ "clip_ratio/low_min": 0.0,
409
+ "clip_ratio/region_mean": 0.0,
410
+ "completions/clipped_ratio": 0.0,
411
+ "completions/max_length": 583.5,
412
+ "completions/max_terminated_length": 583.5,
413
+ "completions/mean_length": 217.625,
414
+ "completions/mean_terminated_length": 217.625,
415
+ "completions/min_length": 37.0,
416
+ "completions/min_terminated_length": 37.0,
417
+ "epoch": 6.797680631368577e-05,
418
+ "frac_reward_zero_std": 1.0,
419
+ "grad_norm": 0.0021501195151358843,
420
+ "kl": 0.0015104188605619129,
421
+ "learning_rate": 4.1647880625292027e-07,
422
+ "loss": 0.0,
423
+ "num_tokens": 78713.0,
424
+ "reward": 0.0,
425
+ "reward_std": 0.0,
426
+ "rewards/equation_reward_func/mean": 0.0,
427
+ "rewards/equation_reward_func/std": 0.0,
428
+ "rewards/format_reward_func/mean": 0.0,
429
+ "rewards/format_reward_func/std": 0.0,
430
+ "step": 30
431
+ },
432
+ {
433
+ "clip_ratio/high_max": 0.0,
434
+ "clip_ratio/high_mean": 0.0,
435
+ "clip_ratio/low_mean": 0.0,
436
+ "clip_ratio/low_min": 0.0,
437
+ "clip_ratio/region_mean": 0.0,
438
+ "completions/clipped_ratio": 0.0,
439
+ "completions/max_length": 335.5,
440
+ "completions/max_terminated_length": 335.5,
441
+ "completions/mean_length": 134.75,
442
+ "completions/mean_terminated_length": 134.75,
443
+ "completions/min_length": 32.0,
444
+ "completions/min_terminated_length": 32.0,
445
+ "epoch": 7.250859340126482e-05,
446
+ "frac_reward_zero_std": 0.0,
447
+ "grad_norm": 14.151634216308594,
448
+ "kl": 0.0026571638591121882,
449
+ "learning_rate": 4.040571164002318e-07,
450
+ "loss": 0.0,
451
+ "num_tokens": 83149.0,
452
+ "reward": 0.1875,
453
+ "reward_std": 0.408231720328331,
454
+ "rewards/equation_reward_func/mean": 0.0,
455
+ "rewards/equation_reward_func/std": 0.0,
456
+ "rewards/format_reward_func/mean": 0.1875,
457
+ "rewards/format_reward_func/std": 0.408231720328331,
458
+ "step": 32
459
+ },
460
+ {
461
+ "clip_ratio/high_max": 0.0,
462
+ "clip_ratio/high_mean": 0.0,
463
+ "clip_ratio/low_mean": 0.0,
464
+ "clip_ratio/low_min": 0.0,
465
+ "clip_ratio/region_mean": 0.0,
466
+ "completions/clipped_ratio": 0.0,
467
+ "completions/max_length": 381.5,
468
+ "completions/max_terminated_length": 381.5,
469
+ "completions/mean_length": 190.625,
470
+ "completions/mean_terminated_length": 190.625,
471
+ "completions/min_length": 34.5,
472
+ "completions/min_terminated_length": 34.5,
473
+ "epoch": 7.704038048884388e-05,
474
+ "frac_reward_zero_std": 1.0,
475
+ "grad_norm": 0.006240386515855789,
476
+ "kl": 0.0022764305977034383,
477
+ "learning_rate": 3.909892574627266e-07,
478
+ "loss": 0.0,
479
+ "num_tokens": 88487.0,
480
+ "reward": 0.0,
481
+ "reward_std": 0.0,
482
+ "rewards/equation_reward_func/mean": 0.0,
483
+ "rewards/equation_reward_func/std": 0.0,
484
+ "rewards/format_reward_func/mean": 0.0,
485
+ "rewards/format_reward_func/std": 0.0,
486
+ "step": 34
487
+ },
488
+ {
489
+ "clip_ratio/high_max": 0.0,
490
+ "clip_ratio/high_mean": 0.0,
491
+ "clip_ratio/low_mean": 0.0,
492
+ "clip_ratio/low_min": 0.0,
493
+ "clip_ratio/region_mean": 0.0,
494
+ "completions/clipped_ratio": 0.0,
495
+ "completions/max_length": 745.0,
496
+ "completions/max_terminated_length": 745.0,
497
+ "completions/mean_length": 219.75,
498
+ "completions/mean_terminated_length": 219.75,
499
+ "completions/min_length": 67.5,
500
+ "completions/min_terminated_length": 67.5,
501
+ "epoch": 8.157216757642292e-05,
502
+ "frac_reward_zero_std": 0.0,
503
+ "grad_norm": 16.2641544342041,
504
+ "kl": 0.002281889770529233,
505
+ "learning_rate": 3.773300405821908e-07,
506
+ "loss": 0.0,
507
+ "num_tokens": 94251.0,
508
+ "reward": 0.125,
509
+ "reward_std": 0.3535533845424652,
510
+ "rewards/equation_reward_func/mean": 0.0,
511
+ "rewards/equation_reward_func/std": 0.0,
512
+ "rewards/format_reward_func/mean": 0.125,
513
+ "rewards/format_reward_func/std": 0.3535533845424652,
514
+ "step": 36
515
+ },
516
+ {
517
+ "clip_ratio/high_max": 0.0,
518
+ "clip_ratio/high_mean": 0.0,
519
+ "clip_ratio/low_mean": 0.0,
520
+ "clip_ratio/low_min": 0.0,
521
+ "clip_ratio/region_mean": 0.0,
522
+ "completions/clipped_ratio": 0.0,
523
+ "completions/max_length": 436.0,
524
+ "completions/max_terminated_length": 436.0,
525
+ "completions/mean_length": 172.5,
526
+ "completions/mean_terminated_length": 172.5,
527
+ "completions/min_length": 66.0,
528
+ "completions/min_terminated_length": 66.0,
529
+ "epoch": 8.610395466400197e-05,
530
+ "frac_reward_zero_std": 0.5,
531
+ "grad_norm": 15.585210800170898,
532
+ "kl": 0.010958031325571937,
533
+ "learning_rate": 3.6313675726113475e-07,
534
+ "loss": 0.0,
535
+ "num_tokens": 99331.0,
536
+ "reward": 0.125,
537
+ "reward_std": 0.2314550280570984,
538
+ "rewards/equation_reward_func/mean": 0.0,
539
+ "rewards/equation_reward_func/std": 0.0,
540
+ "rewards/format_reward_func/mean": 0.125,
541
+ "rewards/format_reward_func/std": 0.2314550280570984,
542
+ "step": 38
543
+ },
544
+ {
545
+ "clip_ratio/high_max": 0.0,
546
+ "clip_ratio/high_mean": 0.0,
547
+ "clip_ratio/low_mean": 0.0,
548
+ "clip_ratio/low_min": 0.0,
549
+ "clip_ratio/region_mean": 0.0,
550
+ "completions/clipped_ratio": 0.0,
551
+ "completions/max_length": 644.5,
552
+ "completions/max_terminated_length": 644.5,
553
+ "completions/mean_length": 238.0,
554
+ "completions/mean_terminated_length": 238.0,
555
+ "completions/min_length": 64.0,
556
+ "completions/min_terminated_length": 64.0,
557
+ "epoch": 9.063574175158103e-05,
558
+ "frac_reward_zero_std": 0.5,
559
+ "grad_norm": 0.0022101891227066517,
560
+ "kl": 0.003324320729007013,
561
+ "learning_rate": 3.484689390623218e-07,
562
+ "loss": 0.0,
563
+ "num_tokens": 105419.0,
564
+ "reward": 0.125,
565
+ "reward_std": 0.2314550280570984,
566
+ "rewards/equation_reward_func/mean": 0.0,
567
+ "rewards/equation_reward_func/std": 0.0,
568
+ "rewards/format_reward_func/mean": 0.125,
569
+ "rewards/format_reward_func/std": 0.2314550280570984,
570
+ "step": 40
571
+ },
572
+ {
573
+ "clip_ratio/high_max": 0.0,
574
+ "clip_ratio/high_mean": 0.0,
575
+ "clip_ratio/low_mean": 0.0,
576
+ "clip_ratio/low_min": 0.0,
577
+ "clip_ratio/region_mean": 0.0,
578
+ "completions/clipped_ratio": 0.0,
579
+ "completions/max_length": 427.0,
580
+ "completions/max_terminated_length": 427.0,
581
+ "completions/mean_length": 203.3125,
582
+ "completions/mean_terminated_length": 203.3125,
583
+ "completions/min_length": 67.0,
584
+ "completions/min_terminated_length": 67.0,
585
+ "epoch": 9.516752883916008e-05,
586
+ "frac_reward_zero_std": 0.5,
587
+ "grad_norm": 16.7834529876709,
588
+ "kl": 0.002571089873526944,
589
+ "learning_rate": 3.3338810791270517e-07,
590
+ "loss": 0.0,
591
+ "num_tokens": 110992.0,
592
+ "reward": 0.0625,
593
+ "reward_std": 0.1767766922712326,
594
+ "rewards/equation_reward_func/mean": 0.0,
595
+ "rewards/equation_reward_func/std": 0.0,
596
+ "rewards/format_reward_func/mean": 0.0625,
597
+ "rewards/format_reward_func/std": 0.1767766922712326,
598
+ "step": 42
599
+ },
600
+ {
601
+ "clip_ratio/high_max": 0.0,
602
+ "clip_ratio/high_mean": 0.0,
603
+ "clip_ratio/low_mean": 0.0,
604
+ "clip_ratio/low_min": 0.0,
605
+ "clip_ratio/region_mean": 0.0,
606
+ "completions/clipped_ratio": 0.0,
607
+ "completions/max_length": 267.5,
608
+ "completions/max_terminated_length": 267.5,
609
+ "completions/mean_length": 119.25,
610
+ "completions/mean_terminated_length": 119.25,
611
+ "completions/min_length": 55.5,
612
+ "completions/min_terminated_length": 55.5,
613
+ "epoch": 9.969931592673912e-05,
614
+ "frac_reward_zero_std": 0.0,
615
+ "grad_norm": 13.29578685760498,
616
+ "kl": 0.0029905991395935416,
617
+ "learning_rate": 3.179575180590857e-07,
618
+ "loss": 0.0,
619
+ "num_tokens": 115204.0,
620
+ "reward": 0.125,
621
+ "reward_std": 0.3535533845424652,
622
+ "rewards/equation_reward_func/mean": 0.0,
623
+ "rewards/equation_reward_func/std": 0.0,
624
+ "rewards/format_reward_func/mean": 0.125,
625
+ "rewards/format_reward_func/std": 0.3535533845424652,
626
+ "step": 44
627
+ },
628
+ {
629
+ "clip_ratio/high_max": 0.0,
630
+ "clip_ratio/high_mean": 0.0,
631
+ "clip_ratio/low_mean": 0.0,
632
+ "clip_ratio/low_min": 0.0,
633
+ "clip_ratio/region_mean": 0.0,
634
+ "completions/clipped_ratio": 0.0,
635
+ "completions/max_length": 326.5,
636
+ "completions/max_terminated_length": 326.5,
637
+ "completions/mean_length": 152.5,
638
+ "completions/mean_terminated_length": 152.5,
639
+ "completions/min_length": 26.0,
640
+ "completions/min_terminated_length": 26.0,
641
+ "epoch": 0.00010423110301431818,
642
+ "frac_reward_zero_std": 0.5,
643
+ "grad_norm": 11.780332565307617,
644
+ "kl": 0.004898008599411696,
645
+ "learning_rate": 3.022418907578188e-07,
646
+ "loss": 0.0,
647
+ "num_tokens": 119916.0,
648
+ "reward": 0.0625,
649
+ "reward_std": 0.1767766922712326,
650
+ "rewards/equation_reward_func/mean": 0.0,
651
+ "rewards/equation_reward_func/std": 0.0,
652
+ "rewards/format_reward_func/mean": 0.0625,
653
+ "rewards/format_reward_func/std": 0.1767766922712326,
654
+ "step": 46
655
+ },
656
+ {
657
+ "clip_ratio/high_max": 0.0,
658
+ "clip_ratio/high_mean": 0.0,
659
+ "clip_ratio/low_mean": 0.0,
660
+ "clip_ratio/low_min": 0.0,
661
+ "clip_ratio/region_mean": 0.0,
662
+ "completions/clipped_ratio": 0.0,
663
+ "completions/max_length": 756.5,
664
+ "completions/max_terminated_length": 756.5,
665
+ "completions/mean_length": 237.625,
666
+ "completions/mean_terminated_length": 237.625,
667
+ "completions/min_length": 35.0,
668
+ "completions/min_terminated_length": 35.0,
669
+ "epoch": 0.00010876289010189723,
670
+ "frac_reward_zero_std": 0.0,
671
+ "grad_norm": 14.306221008300781,
672
+ "kl": 0.004835129271668848,
673
+ "learning_rate": 2.863071428113726e-07,
674
+ "loss": 0.0,
675
+ "num_tokens": 125990.0,
676
+ "reward": 0.1875,
677
+ "reward_std": 0.408231720328331,
678
+ "rewards/equation_reward_func/mean": 0.0,
679
+ "rewards/equation_reward_func/std": 0.0,
680
+ "rewards/format_reward_func/mean": 0.1875,
681
+ "rewards/format_reward_func/std": 0.408231720328331,
682
+ "step": 48
683
+ },
684
+ {
685
+ "clip_ratio/high_max": 0.0,
686
+ "clip_ratio/high_mean": 0.0,
687
+ "clip_ratio/low_mean": 0.0,
688
+ "clip_ratio/low_min": 0.0,
689
+ "clip_ratio/region_mean": 0.0,
690
+ "completions/clipped_ratio": 0.0,
691
+ "completions/max_length": 405.5,
692
+ "completions/max_terminated_length": 405.5,
693
+ "completions/mean_length": 189.9375,
694
+ "completions/mean_terminated_length": 189.9375,
695
+ "completions/min_length": 67.0,
696
+ "completions/min_terminated_length": 67.0,
697
+ "epoch": 0.00011329467718947628,
698
+ "frac_reward_zero_std": 0.0,
699
+ "grad_norm": 10.71300983428955,
700
+ "kl": 0.006452581874327734,
701
+ "learning_rate": 2.7022011009035107e-07,
702
+ "loss": 0.0,
703
+ "num_tokens": 131301.0,
704
+ "reward": 0.125,
705
+ "reward_std": 0.3535533845424652,
706
+ "rewards/equation_reward_func/mean": 0.0,
707
+ "rewards/equation_reward_func/std": 0.0,
708
+ "rewards/format_reward_func/mean": 0.125,
709
+ "rewards/format_reward_func/std": 0.3535533845424652,
710
+ "step": 50
711
+ },
712
+ {
713
+ "clip_ratio/high_max": 0.0,
714
+ "clip_ratio/high_mean": 0.0,
715
+ "clip_ratio/low_mean": 0.0,
716
+ "clip_ratio/low_min": 0.0,
717
+ "clip_ratio/region_mean": 0.0,
718
+ "completions/clipped_ratio": 0.0,
719
+ "completions/max_length": 361.0,
720
+ "completions/max_terminated_length": 361.0,
721
+ "completions/mean_length": 157.0625,
722
+ "completions/mean_terminated_length": 157.0625,
723
+ "completions/min_length": 59.0,
724
+ "completions/min_terminated_length": 59.0,
725
+ "epoch": 0.00011782646427705534,
726
+ "frac_reward_zero_std": 0.5,
727
+ "grad_norm": 11.726592063903809,
728
+ "kl": 0.0094611946187797,
729
+ "learning_rate": 2.540482672006254e-07,
730
+ "loss": 0.0,
731
+ "num_tokens": 136102.0,
732
+ "reward": 0.25,
733
+ "reward_std": 0.26726123690605164,
734
+ "rewards/equation_reward_func/mean": 0.0,
735
+ "rewards/equation_reward_func/std": 0.0,
736
+ "rewards/format_reward_func/mean": 0.25,
737
+ "rewards/format_reward_func/std": 0.26726123690605164,
738
+ "step": 52
739
+ },
740
+ {
741
+ "clip_ratio/high_max": 0.0,
742
+ "clip_ratio/high_mean": 0.0,
743
+ "clip_ratio/low_mean": 0.0,
744
+ "clip_ratio/low_min": 0.0,
745
+ "clip_ratio/region_mean": 0.0,
746
+ "completions/clipped_ratio": 0.0,
747
+ "completions/max_length": 213.0,
748
+ "completions/max_terminated_length": 213.0,
749
+ "completions/mean_length": 117.6875,
750
+ "completions/mean_terminated_length": 117.6875,
751
+ "completions/min_length": 44.5,
752
+ "completions/min_terminated_length": 44.5,
753
+ "epoch": 0.00012235825136463439,
754
+ "frac_reward_zero_std": 0.5,
755
+ "grad_norm": 13.030102729797363,
756
+ "kl": 0.006068318209145218,
757
+ "learning_rate": 2.37859444471388e-07,
758
+ "loss": 0.0,
759
+ "num_tokens": 140241.0,
760
+ "reward": 0.0625,
761
+ "reward_std": 0.1767766922712326,
762
+ "rewards/equation_reward_func/mean": 0.0,
763
+ "rewards/equation_reward_func/std": 0.0,
764
+ "rewards/format_reward_func/mean": 0.0625,
765
+ "rewards/format_reward_func/std": 0.1767766922712326,
766
+ "step": 54
767
+ },
768
+ {
769
+ "clip_ratio/high_max": 0.0,
770
+ "clip_ratio/high_mean": 0.0,
771
+ "clip_ratio/low_mean": 0.0,
772
+ "clip_ratio/low_min": 0.0,
773
+ "clip_ratio/region_mean": 0.0,
774
+ "completions/clipped_ratio": 0.0,
775
+ "completions/max_length": 333.0,
776
+ "completions/max_terminated_length": 333.0,
777
+ "completions/mean_length": 136.25,
778
+ "completions/mean_terminated_length": 136.25,
779
+ "completions/min_length": 36.0,
780
+ "completions/min_terminated_length": 36.0,
781
+ "epoch": 0.00012689003845221345,
782
+ "frac_reward_zero_std": 0.0,
783
+ "grad_norm": 10.891292572021484,
784
+ "kl": 0.009872130773146637,
785
+ "learning_rate": 2.2172154345117894e-07,
786
+ "loss": 0.0,
787
+ "num_tokens": 144701.0,
788
+ "reward": 0.25,
789
+ "reward_std": 0.4355512708425522,
790
+ "rewards/equation_reward_func/mean": 0.0,
791
+ "rewards/equation_reward_func/std": 0.0,
792
+ "rewards/format_reward_func/mean": 0.25,
793
+ "rewards/format_reward_func/std": 0.4355513006448746,
794
+ "step": 56
795
+ },
796
+ {
797
+ "clip_ratio/high_max": 0.0,
798
+ "clip_ratio/high_mean": 0.0,
799
+ "clip_ratio/low_mean": 0.0,
800
+ "clip_ratio/low_min": 0.0,
801
+ "clip_ratio/region_mean": 0.0,
802
+ "completions/clipped_ratio": 0.0,
803
+ "completions/max_length": 663.5,
804
+ "completions/max_terminated_length": 663.5,
805
+ "completions/mean_length": 196.625,
806
+ "completions/mean_terminated_length": 196.625,
807
+ "completions/min_length": 59.5,
808
+ "completions/min_terminated_length": 59.5,
809
+ "epoch": 0.00013142182553979248,
810
+ "frac_reward_zero_std": 0.0,
811
+ "grad_norm": 11.184815406799316,
812
+ "kl": 0.01285637664841488,
813
+ "learning_rate": 2.0570225210519433e-07,
814
+ "loss": 0.0,
815
+ "num_tokens": 150159.0,
816
+ "reward": 0.1875,
817
+ "reward_std": 0.408231720328331,
818
+ "rewards/equation_reward_func/mean": 0.0,
819
+ "rewards/equation_reward_func/std": 0.0,
820
+ "rewards/format_reward_func/mean": 0.1875,
821
+ "rewards/format_reward_func/std": 0.408231720328331,
822
+ "step": 58
823
+ },
824
+ {
825
+ "clip_ratio/high_max": 0.0,
826
+ "clip_ratio/high_mean": 0.0,
827
+ "clip_ratio/low_mean": 0.0,
828
+ "clip_ratio/low_min": 0.0,
829
+ "clip_ratio/region_mean": 0.0,
830
+ "completions/clipped_ratio": 0.0,
831
+ "completions/max_length": 308.0,
832
+ "completions/max_terminated_length": 308.0,
833
+ "completions/mean_length": 140.4375,
834
+ "completions/mean_terminated_length": 140.4375,
835
+ "completions/min_length": 50.0,
836
+ "completions/min_terminated_length": 50.0,
837
+ "epoch": 0.00013595361262737154,
838
+ "frac_reward_zero_std": 0.5,
839
+ "grad_norm": 0.02632570080459118,
840
+ "kl": 0.014614543215429876,
841
+ "learning_rate": 1.8986876090843664e-07,
842
+ "loss": 0.0,
843
+ "num_tokens": 154654.0,
844
+ "reward": 0.1875,
845
+ "reward_std": 0.2587745785713196,
846
+ "rewards/equation_reward_func/mean": 0.0,
847
+ "rewards/equation_reward_func/std": 0.0,
848
+ "rewards/format_reward_func/mean": 0.1875,
849
+ "rewards/format_reward_func/std": 0.25877460837364197,
850
+ "step": 60
851
+ },
852
+ {
853
+ "clip_ratio/high_max": 0.0,
854
+ "clip_ratio/high_mean": 0.0,
855
+ "clip_ratio/low_mean": 0.0,
856
+ "clip_ratio/low_min": 0.0,
857
+ "clip_ratio/region_mean": 0.0,
858
+ "completions/clipped_ratio": 0.0,
859
+ "completions/max_length": 270.0,
860
+ "completions/max_terminated_length": 270.0,
861
+ "completions/mean_length": 104.0625,
862
+ "completions/mean_terminated_length": 104.0625,
863
+ "completions/min_length": 38.0,
864
+ "completions/min_terminated_length": 38.0,
865
+ "epoch": 0.0001404853997149506,
866
+ "frac_reward_zero_std": 0.0,
867
+ "grad_norm": 11.164044380187988,
868
+ "kl": 0.007626835664268583,
869
+ "learning_rate": 1.7428748102551234e-07,
870
+ "loss": 0.0,
871
+ "num_tokens": 158599.0,
872
+ "reward": 0.625,
873
+ "reward_std": 0.49871626496315,
874
+ "rewards/equation_reward_func/mean": 0.0,
875
+ "rewards/equation_reward_func/std": 0.0,
876
+ "rewards/format_reward_func/mean": 0.625,
877
+ "rewards/format_reward_func/std": 0.49871626496315,
878
+ "step": 62
879
+ },
880
+ {
881
+ "clip_ratio/high_max": 0.0,
882
+ "clip_ratio/high_mean": 0.0,
883
+ "clip_ratio/low_mean": 0.0,
884
+ "clip_ratio/low_min": 0.0,
885
+ "clip_ratio/region_mean": 0.0,
886
+ "completions/clipped_ratio": 0.0,
887
+ "completions/max_length": 160.5,
888
+ "completions/max_terminated_length": 160.5,
889
+ "completions/mean_length": 90.625,
890
+ "completions/mean_terminated_length": 90.625,
891
+ "completions/min_length": 47.0,
892
+ "completions/min_terminated_length": 47.0,
893
+ "epoch": 0.00014501718680252963,
894
+ "frac_reward_zero_std": 0.0,
895
+ "grad_norm": 16.527629852294922,
896
+ "kl": 0.011474673578049988,
897
+ "learning_rate": 1.5902376575912814e-07,
898
+ "loss": 0.0,
899
+ "num_tokens": 162289.0,
900
+ "reward": 0.3125,
901
+ "reward_std": 0.44403792917728424,
902
+ "rewards/equation_reward_func/mean": 0.0,
903
+ "rewards/equation_reward_func/std": 0.0,
904
+ "rewards/format_reward_func/mean": 0.3125,
905
+ "rewards/format_reward_func/std": 0.44403792917728424,
906
+ "step": 64
907
+ },
908
+ {
909
+ "clip_ratio/high_max": 0.0,
910
+ "clip_ratio/high_mean": 0.0,
911
+ "clip_ratio/low_mean": 0.0,
912
+ "clip_ratio/low_min": 0.0,
913
+ "clip_ratio/region_mean": 0.0,
914
+ "completions/clipped_ratio": 0.0,
915
+ "completions/max_length": 398.5,
916
+ "completions/max_terminated_length": 398.5,
917
+ "completions/mean_length": 129.0625,
918
+ "completions/mean_terminated_length": 129.0625,
919
+ "completions/min_length": 47.0,
920
+ "completions/min_terminated_length": 47.0,
921
+ "epoch": 0.0001495489738901087,
922
+ "frac_reward_zero_std": 0.0,
923
+ "grad_norm": 12.219941139221191,
924
+ "kl": 0.009613552174414508,
925
+ "learning_rate": 1.4414163643562753e-07,
926
+ "loss": 0.0,
927
+ "num_tokens": 166674.0,
928
+ "reward": 0.25,
929
+ "reward_std": 0.4355512708425522,
930
+ "rewards/equation_reward_func/mean": 0.0,
931
+ "rewards/equation_reward_func/std": 0.0,
932
+ "rewards/format_reward_func/mean": 0.25,
933
+ "rewards/format_reward_func/std": 0.4355513006448746,
934
+ "step": 66
935
+ },
936
+ {
937
+ "clip_ratio/high_max": 0.0,
938
+ "clip_ratio/high_mean": 0.0,
939
+ "clip_ratio/low_mean": 0.0,
940
+ "clip_ratio/low_min": 0.0,
941
+ "clip_ratio/region_mean": 0.0,
942
+ "completions/clipped_ratio": 0.0,
943
+ "completions/max_length": 244.5,
944
+ "completions/max_terminated_length": 244.5,
945
+ "completions/mean_length": 109.1875,
946
+ "completions/mean_terminated_length": 109.1875,
947
+ "completions/min_length": 45.5,
948
+ "completions/min_terminated_length": 45.5,
949
+ "epoch": 0.00015408076097768775,
950
+ "frac_reward_zero_std": 0.0,
951
+ "grad_norm": 20.217954635620117,
952
+ "kl": 0.020021719275973737,
953
+ "learning_rate": 1.2970351387729872e-07,
954
+ "loss": 0.0,
955
+ "num_tokens": 170693.0,
956
+ "reward": 0.375,
957
+ "reward_std": 0.5175491571426392,
958
+ "rewards/equation_reward_func/mean": 0.0,
959
+ "rewards/equation_reward_func/std": 0.0,
960
+ "rewards/format_reward_func/mean": 0.375,
961
+ "rewards/format_reward_func/std": 0.5175492167472839,
962
+ "step": 68
963
+ },
964
+ {
965
+ "clip_ratio/high_max": 0.0,
966
+ "clip_ratio/high_mean": 0.0,
967
+ "clip_ratio/low_mean": 0.0,
968
+ "clip_ratio/low_min": 0.0,
969
+ "clip_ratio/region_mean": 0.0,
970
+ "completions/clipped_ratio": 0.0,
971
+ "completions/max_length": 329.0,
972
+ "completions/max_terminated_length": 329.0,
973
+ "completions/mean_length": 127.0625,
974
+ "completions/mean_terminated_length": 127.0625,
975
+ "completions/min_length": 45.0,
976
+ "completions/min_terminated_length": 45.0,
977
+ "epoch": 0.0001586125480652668,
978
+ "frac_reward_zero_std": 0.0,
979
+ "grad_norm": 12.027848243713379,
980
+ "kl": 0.01639675306796562,
981
+ "learning_rate": 1.1576995658775404e-07,
982
+ "loss": 0.0,
983
+ "num_tokens": 175014.0,
984
+ "reward": 0.4375,
985
+ "reward_std": 0.5260358154773712,
986
+ "rewards/equation_reward_func/mean": 0.0,
987
+ "rewards/equation_reward_func/std": 0.0,
988
+ "rewards/format_reward_func/mean": 0.4375,
989
+ "rewards/format_reward_func/std": 0.5260358452796936,
990
+ "step": 70
991
+ },
992
+ {
993
+ "clip_ratio/high_max": 0.0,
994
+ "clip_ratio/high_mean": 0.0,
995
+ "clip_ratio/low_mean": 0.0,
996
+ "clip_ratio/low_min": 0.0,
997
+ "clip_ratio/region_mean": 0.0,
998
+ "completions/clipped_ratio": 0.0,
999
+ "completions/max_length": 375.0,
1000
+ "completions/max_terminated_length": 375.0,
1001
+ "completions/mean_length": 142.1875,
1002
+ "completions/mean_terminated_length": 142.1875,
1003
+ "completions/min_length": 52.0,
1004
+ "completions/min_terminated_length": 52.0,
1005
+ "epoch": 0.00016314433515284585,
1006
+ "frac_reward_zero_std": 0.0,
1007
+ "grad_norm": 10.927833557128906,
1008
+ "kl": 0.011894080380443484,
1009
+ "learning_rate": 1.0239940674851941e-07,
1010
+ "loss": 0.0,
1011
+ "num_tokens": 179577.0,
1012
+ "reward": 0.25,
1013
+ "reward_std": 0.4629100561141968,
1014
+ "rewards/equation_reward_func/mean": 0.0,
1015
+ "rewards/equation_reward_func/std": 0.0,
1016
+ "rewards/format_reward_func/mean": 0.25,
1017
+ "rewards/format_reward_func/std": 0.4629100561141968,
1018
+ "step": 72
1019
+ },
1020
+ {
1021
+ "clip_ratio/high_max": 0.0,
1022
+ "clip_ratio/high_mean": 0.0,
1023
+ "clip_ratio/low_mean": 0.0,
1024
+ "clip_ratio/low_min": 0.0,
1025
+ "clip_ratio/region_mean": 0.0,
1026
+ "completions/clipped_ratio": 0.0,
1027
+ "completions/max_length": 951.0,
1028
+ "completions/max_terminated_length": 951.0,
1029
+ "completions/mean_length": 273.6875,
1030
+ "completions/mean_terminated_length": 273.6875,
1031
+ "completions/min_length": 41.5,
1032
+ "completions/min_terminated_length": 41.5,
1033
+ "epoch": 0.0001676761222404249,
1034
+ "frac_reward_zero_std": 0.0,
1035
+ "grad_norm": 6.783898830413818,
1036
+ "kl": 0.014139190083369613,
1037
+ "learning_rate": 8.964794509221507e-08,
1038
+ "loss": 0.0,
1039
+ "num_tokens": 186236.0,
1040
+ "reward": 0.375,
1041
+ "reward_std": 0.4355512708425522,
1042
+ "rewards/equation_reward_func/mean": 0.0,
1043
+ "rewards/equation_reward_func/std": 0.0,
1044
+ "rewards/format_reward_func/mean": 0.375,
1045
+ "rewards/format_reward_func/std": 0.4355513006448746,
1046
+ "step": 74
1047
+ },
1048
+ {
1049
+ "clip_ratio/high_max": 0.0,
1050
+ "clip_ratio/high_mean": 0.0,
1051
+ "clip_ratio/low_mean": 0.0,
1052
+ "clip_ratio/low_min": 0.0,
1053
+ "clip_ratio/region_mean": 0.0,
1054
+ "completions/clipped_ratio": 0.0,
1055
+ "completions/max_length": 174.5,
1056
+ "completions/max_terminated_length": 174.5,
1057
+ "completions/mean_length": 106.25,
1058
+ "completions/mean_terminated_length": 106.25,
1059
+ "completions/min_length": 44.0,
1060
+ "completions/min_terminated_length": 44.0,
1061
+ "epoch": 0.00017220790932800394,
1062
+ "frac_reward_zero_std": 0.0,
1063
+ "grad_norm": 12.524798393249512,
1064
+ "kl": 0.013552291362429969,
1065
+ "learning_rate": 7.756905568047392e-08,
1066
+ "loss": 0.0,
1067
+ "num_tokens": 190168.0,
1068
+ "reward": 0.5625,
1069
+ "reward_std": 0.5260358154773712,
1070
+ "rewards/equation_reward_func/mean": 0.0,
1071
+ "rewards/equation_reward_func/std": 0.0,
1072
+ "rewards/format_reward_func/mean": 0.5625,
1073
+ "rewards/format_reward_func/std": 0.5260358452796936,
1074
+ "step": 76
1075
+ },
1076
+ {
1077
+ "clip_ratio/high_max": 0.0,
1078
+ "clip_ratio/high_mean": 0.0,
1079
+ "clip_ratio/low_mean": 0.0,
1080
+ "clip_ratio/low_min": 0.0,
1081
+ "clip_ratio/region_mean": 0.0,
1082
+ "completions/clipped_ratio": 0.0,
1083
+ "completions/max_length": 647.5,
1084
+ "completions/max_terminated_length": 647.5,
1085
+ "completions/mean_length": 207.5625,
1086
+ "completions/mean_terminated_length": 207.5625,
1087
+ "completions/min_length": 46.0,
1088
+ "completions/min_terminated_length": 46.0,
1089
+ "epoch": 0.000176739696415583,
1090
+ "frac_reward_zero_std": 0.0,
1091
+ "grad_norm": 10.13427448272705,
1092
+ "kl": 0.01421075320104137,
1093
+ "learning_rate": 6.621340157319996e-08,
1094
+ "loss": 0.0,
1095
+ "num_tokens": 195793.0,
1096
+ "reward": 0.375,
1097
+ "reward_std": 0.4355512708425522,
1098
+ "rewards/equation_reward_func/mean": 0.0,
1099
+ "rewards/equation_reward_func/std": 0.0,
1100
+ "rewards/format_reward_func/mean": 0.375,
1101
+ "rewards/format_reward_func/std": 0.4355513006448746,
1102
+ "step": 78
1103
+ },
1104
+ {
1105
+ "clip_ratio/high_max": 0.0,
1106
+ "clip_ratio/high_mean": 0.0,
1107
+ "clip_ratio/low_mean": 0.0,
1108
+ "clip_ratio/low_min": 0.0,
1109
+ "clip_ratio/region_mean": 0.0,
1110
+ "completions/clipped_ratio": 0.0625,
1111
+ "completions/max_length": 620.0,
1112
+ "completions/max_terminated_length": 281.0,
1113
+ "completions/mean_length": 187.5625,
1114
+ "completions/mean_terminated_length": 134.85714721679688,
1115
+ "completions/min_length": 40.0,
1116
+ "completions/min_terminated_length": 40.0,
1117
+ "epoch": 0.00018127148350316206,
1118
+ "frac_reward_zero_std": 0.0,
1119
+ "grad_norm": 16.1104793548584,
1120
+ "kl": 0.014304678879852872,
1121
+ "learning_rate": 5.5628612330087724e-08,
1122
+ "loss": 0.0,
1123
+ "num_tokens": 201050.0,
1124
+ "reward": 0.25,
1125
+ "reward_std": 0.4355512708425522,
1126
+ "rewards/equation_reward_func/mean": 0.0,
1127
+ "rewards/equation_reward_func/std": 0.0,
1128
+ "rewards/format_reward_func/mean": 0.25,
1129
+ "rewards/format_reward_func/std": 0.4355513006448746,
1130
+ "step": 80
1131
+ },
1132
+ {
1133
+ "clip_ratio/high_max": 0.0,
1134
+ "clip_ratio/high_mean": 0.0,
1135
+ "clip_ratio/low_mean": 0.0,
1136
+ "clip_ratio/low_min": 0.0,
1137
+ "clip_ratio/region_mean": 0.0,
1138
+ "completions/clipped_ratio": 0.0625,
1139
+ "completions/max_length": 746.0,
1140
+ "completions/max_terminated_length": 345.5,
1141
+ "completions/mean_length": 174.6875,
1142
+ "completions/mean_terminated_length": 116.39286041259766,
1143
+ "completions/min_length": 50.5,
1144
+ "completions/min_terminated_length": 50.5,
1145
+ "epoch": 0.0001858032705907411,
1146
+ "frac_reward_zero_std": 0.0,
1147
+ "grad_norm": 13.687472343444824,
1148
+ "kl": 0.012647167037357576,
1149
+ "learning_rate": 4.5859084235697235e-08,
1150
+ "loss": 0.0,
1151
+ "num_tokens": 206125.0,
1152
+ "reward": 0.25,
1153
+ "reward_std": 0.4629100561141968,
1154
+ "rewards/equation_reward_func/mean": 0.0,
1155
+ "rewards/equation_reward_func/std": 0.0,
1156
+ "rewards/format_reward_func/mean": 0.25,
1157
+ "rewards/format_reward_func/std": 0.4629100561141968,
1158
+ "step": 82
1159
+ },
1160
+ {
1161
+ "clip_ratio/high_max": 0.0,
1162
+ "clip_ratio/high_mean": 0.0,
1163
+ "clip_ratio/low_mean": 0.0,
1164
+ "clip_ratio/low_min": 0.0,
1165
+ "clip_ratio/region_mean": 0.0,
1166
+ "completions/clipped_ratio": 0.0,
1167
+ "completions/max_length": 240.5,
1168
+ "completions/max_terminated_length": 240.5,
1169
+ "completions/mean_length": 104.75,
1170
+ "completions/mean_terminated_length": 104.75,
1171
+ "completions/min_length": 56.0,
1172
+ "completions/min_terminated_length": 56.0,
1173
+ "epoch": 0.00019033505767832015,
1174
+ "frac_reward_zero_std": 0.0,
1175
+ "grad_norm": 27.33205795288086,
1176
+ "kl": 0.01843659658334218,
1177
+ "learning_rate": 3.6945794086007705e-08,
1178
+ "loss": 0.0,
1179
+ "num_tokens": 210097.0,
1180
+ "reward": 0.625,
1181
+ "reward_std": 0.49871626496315,
1182
+ "rewards/equation_reward_func/mean": 0.0,
1183
+ "rewards/equation_reward_func/std": 0.0,
1184
+ "rewards/format_reward_func/mean": 0.625,
1185
+ "rewards/format_reward_func/std": 0.49871626496315,
1186
+ "step": 84
1187
+ },
1188
+ {
1189
+ "clip_ratio/high_max": 0.0,
1190
+ "clip_ratio/high_mean": 0.0,
1191
+ "clip_ratio/low_mean": 0.0,
1192
+ "clip_ratio/low_min": 0.0,
1193
+ "clip_ratio/region_mean": 0.0,
1194
+ "completions/clipped_ratio": 0.0,
1195
+ "completions/max_length": 516.5,
1196
+ "completions/max_terminated_length": 516.5,
1197
+ "completions/mean_length": 146.9375,
1198
+ "completions/mean_terminated_length": 146.9375,
1199
+ "completions/min_length": 45.0,
1200
+ "completions/min_terminated_length": 45.0,
1201
+ "epoch": 0.00019486684476589921,
1202
+ "frac_reward_zero_std": 0.0,
1203
+ "grad_norm": 17.247251510620117,
1204
+ "kl": 0.014375057930010371,
1205
+ "learning_rate": 2.892612731749414e-08,
1206
+ "loss": 0.0,
1207
+ "num_tokens": 214696.0,
1208
+ "reward": 0.375,
1209
+ "reward_std": 0.5175491571426392,
1210
+ "rewards/equation_reward_func/mean": 0.0,
1211
+ "rewards/equation_reward_func/std": 0.0,
1212
+ "rewards/format_reward_func/mean": 0.375,
1213
+ "rewards/format_reward_func/std": 0.5175492167472839,
1214
+ "step": 86
1215
+ },
1216
+ {
1217
+ "clip_ratio/high_max": 0.0,
1218
+ "clip_ratio/high_mean": 0.0,
1219
+ "clip_ratio/low_mean": 0.0,
1220
+ "clip_ratio/low_min": 0.0,
1221
+ "clip_ratio/region_mean": 0.0,
1222
+ "completions/clipped_ratio": 0.0,
1223
+ "completions/max_length": 176.0,
1224
+ "completions/max_terminated_length": 176.0,
1225
+ "completions/mean_length": 92.375,
1226
+ "completions/mean_terminated_length": 92.375,
1227
+ "completions/min_length": 39.0,
1228
+ "completions/min_terminated_length": 39.0,
1229
+ "epoch": 0.00019939863185347825,
1230
+ "frac_reward_zero_std": 0.0,
1231
+ "grad_norm": 12.001086235046387,
1232
+ "kl": 0.015669465501559898,
1233
+ "learning_rate": 2.183372119961499e-08,
1234
+ "loss": 0.0,
1235
+ "num_tokens": 218470.0,
1236
+ "reward": 0.1875,
1237
+ "reward_std": 0.408231720328331,
1238
+ "rewards/equation_reward_func/mean": 0.0,
1239
+ "rewards/equation_reward_func/std": 0.0,
1240
+ "rewards/format_reward_func/mean": 0.1875,
1241
+ "rewards/format_reward_func/std": 0.408231720328331,
1242
+ "step": 88
1243
+ },
1244
+ {
1245
+ "clip_ratio/high_max": 0.0,
1246
+ "clip_ratio/high_mean": 0.0,
1247
+ "clip_ratio/low_mean": 0.0,
1248
+ "clip_ratio/low_min": 0.0,
1249
+ "clip_ratio/region_mean": 0.0,
1250
+ "completions/clipped_ratio": 0.0,
1251
+ "completions/max_length": 522.0,
1252
+ "completions/max_terminated_length": 522.0,
1253
+ "completions/mean_length": 174.0,
1254
+ "completions/mean_terminated_length": 174.0,
1255
+ "completions/min_length": 55.5,
1256
+ "completions/min_terminated_length": 55.5,
1257
+ "epoch": 0.0002039304189410573,
1258
+ "frac_reward_zero_std": 0.0,
1259
+ "grad_norm": 11.844170570373535,
1260
+ "kl": 0.009546459768898785,
1261
+ "learning_rate": 1.5698323748414122e-08,
1262
+ "loss": 0.0,
1263
+ "num_tokens": 223534.0,
1264
+ "reward": 0.375,
1265
+ "reward_std": 0.4355512708425522,
1266
+ "rewards/equation_reward_func/mean": 0.0,
1267
+ "rewards/equation_reward_func/std": 0.0,
1268
+ "rewards/format_reward_func/mean": 0.375,
1269
+ "rewards/format_reward_func/std": 0.4355513006448746,
1270
+ "step": 90
1271
+ },
1272
+ {
1273
+ "clip_ratio/high_max": 0.0,
1274
+ "clip_ratio/high_mean": 0.0,
1275
+ "clip_ratio/low_mean": 0.0,
1276
+ "clip_ratio/low_min": 0.0,
1277
+ "clip_ratio/region_mean": 0.0,
1278
+ "completions/clipped_ratio": 0.0,
1279
+ "completions/max_length": 331.5,
1280
+ "completions/max_terminated_length": 331.5,
1281
+ "completions/mean_length": 151.8125,
1282
+ "completions/mean_terminated_length": 151.8125,
1283
+ "completions/min_length": 48.5,
1284
+ "completions/min_terminated_length": 48.5,
1285
+ "epoch": 0.00020846220602863637,
1286
+ "frac_reward_zero_std": 0.0,
1287
+ "grad_norm": 11.816337585449219,
1288
+ "kl": 0.01593888070783578,
1289
+ "learning_rate": 1.054566895300324e-08,
1290
+ "loss": 0.0,
1291
+ "num_tokens": 228243.0,
1292
+ "reward": 0.5625,
1293
+ "reward_std": 0.5260358154773712,
1294
+ "rewards/equation_reward_func/mean": 0.0,
1295
+ "rewards/equation_reward_func/std": 0.0,
1296
+ "rewards/format_reward_func/mean": 0.5625,
1297
+ "rewards/format_reward_func/std": 0.5260358452796936,
1298
+ "step": 92
1299
+ },
1300
+ {
1301
+ "clip_ratio/high_max": 0.0,
1302
+ "clip_ratio/high_mean": 0.0,
1303
+ "clip_ratio/low_mean": 0.0,
1304
+ "clip_ratio/low_min": 0.0,
1305
+ "clip_ratio/region_mean": 0.0,
1306
+ "completions/clipped_ratio": 0.0,
1307
+ "completions/max_length": 387.0,
1308
+ "completions/max_terminated_length": 387.0,
1309
+ "completions/mean_length": 163.6875,
1310
+ "completions/mean_terminated_length": 163.6875,
1311
+ "completions/min_length": 33.0,
1312
+ "completions/min_terminated_length": 33.0,
1313
+ "epoch": 0.0002129939931162154,
1314
+ "frac_reward_zero_std": 0.0,
1315
+ "grad_norm": 11.730420112609863,
1316
+ "kl": 0.01412112163961865,
1317
+ "learning_rate": 6.397368838268496e-09,
1318
+ "loss": 0.0,
1319
+ "num_tokens": 233110.0,
1320
+ "reward": 0.1875,
1321
+ "reward_std": 0.408231720328331,
1322
+ "rewards/equation_reward_func/mean": 0.0,
1323
+ "rewards/equation_reward_func/std": 0.0,
1324
+ "rewards/format_reward_func/mean": 0.1875,
1325
+ "rewards/format_reward_func/std": 0.408231720328331,
1326
+ "step": 94
1327
+ },
1328
+ {
1329
+ "clip_ratio/high_max": 0.0,
1330
+ "clip_ratio/high_mean": 0.0,
1331
+ "clip_ratio/low_mean": 0.0,
1332
+ "clip_ratio/low_min": 0.0,
1333
+ "clip_ratio/region_mean": 0.0,
1334
+ "completions/clipped_ratio": 0.0,
1335
+ "completions/max_length": 420.0,
1336
+ "completions/max_terminated_length": 420.0,
1337
+ "completions/mean_length": 192.4375,
1338
+ "completions/mean_terminated_length": 192.4375,
1339
+ "completions/min_length": 50.0,
1340
+ "completions/min_terminated_length": 50.0,
1341
+ "epoch": 0.00021752578020379446,
1342
+ "frac_reward_zero_std": 0.5,
1343
+ "grad_norm": 15.307268142700195,
1344
+ "kl": 0.00981484999647364,
1345
+ "learning_rate": 3.2708228165273244e-09,
1346
+ "loss": 0.0,
1347
+ "num_tokens": 238493.0,
1348
+ "reward": 0.0625,
1349
+ "reward_std": 0.1767766922712326,
1350
+ "rewards/equation_reward_func/mean": 0.0,
1351
+ "rewards/equation_reward_func/std": 0.0,
1352
+ "rewards/format_reward_func/mean": 0.0625,
1353
+ "rewards/format_reward_func/std": 0.1767766922712326,
1354
+ "step": 96
1355
+ },
1356
+ {
1357
+ "clip_ratio/high_max": 0.0,
1358
+ "clip_ratio/high_mean": 0.0,
1359
+ "clip_ratio/low_mean": 0.0,
1360
+ "clip_ratio/low_min": 0.0,
1361
+ "clip_ratio/region_mean": 0.0,
1362
+ "completions/clipped_ratio": 0.0,
1363
+ "completions/max_length": 372.5,
1364
+ "completions/max_terminated_length": 372.5,
1365
+ "completions/mean_length": 108.1875,
1366
+ "completions/mean_terminated_length": 108.1875,
1367
+ "completions/min_length": 37.0,
1368
+ "completions/min_terminated_length": 37.0,
1369
+ "epoch": 0.00022205756729137352,
1370
+ "frac_reward_zero_std": 0.0,
1371
+ "grad_norm": 23.4942684173584,
1372
+ "kl": 0.022864260390633717,
1373
+ "learning_rate": 1.1791447083465133e-09,
1374
+ "loss": 0.0,
1375
+ "num_tokens": 242536.0,
1376
+ "reward": 0.25,
1377
+ "reward_std": 0.4629100561141968,
1378
+ "rewards/equation_reward_func/mean": 0.0,
1379
+ "rewards/equation_reward_func/std": 0.0,
1380
+ "rewards/format_reward_func/mean": 0.25,
1381
+ "rewards/format_reward_func/std": 0.4629100561141968,
1382
+ "step": 98
1383
+ },
1384
+ {
1385
+ "clip_ratio/high_max": 0.0,
1386
+ "clip_ratio/high_mean": 0.0,
1387
+ "clip_ratio/low_mean": 0.0,
1388
+ "clip_ratio/low_min": 0.0,
1389
+ "clip_ratio/region_mean": 0.0,
1390
+ "completions/clipped_ratio": 0.0,
1391
+ "completions/max_length": 426.0,
1392
+ "completions/max_terminated_length": 426.0,
1393
+ "completions/mean_length": 194.3125,
1394
+ "completions/mean_terminated_length": 194.3125,
1395
+ "completions/min_length": 37.0,
1396
+ "completions/min_terminated_length": 37.0,
1397
+ "epoch": 0.00022658935437895256,
1398
+ "frac_reward_zero_std": 0.0,
1399
+ "grad_norm": 23.61033058166504,
1400
+ "kl": 0.0231838297622744,
1401
+ "learning_rate": 1.3110773862126667e-10,
1402
+ "loss": 0.0,
1403
+ "num_tokens": 247917.0,
1404
+ "reward": 0.4375,
1405
+ "reward_std": 0.5260358154773712,
1406
+ "rewards/equation_reward_func/mean": 0.0,
1407
+ "rewards/equation_reward_func/std": 0.0,
1408
+ "rewards/format_reward_func/mean": 0.4375,
1409
+ "rewards/format_reward_func/std": 0.5260358452796936,
1410
+ "step": 100
1411
+ },
1412
+ {
1413
+ "epoch": 0.00022658935437895256,
1414
+ "step": 100,
1415
+ "total_flos": 0.0,
1416
+ "train_loss": 8.04626297735922e-06,
1417
+ "train_runtime": 2640.6632,
1418
+ "train_samples_per_second": 0.303,
1419
+ "train_steps_per_second": 0.038
1420
+ }
1421
+ ],
1422
+ "logging_steps": 2,
1423
+ "max_steps": 100,
1424
+ "num_input_tokens_seen": 247917,
1425
+ "num_train_epochs": 1,
1426
+ "save_steps": 25,
1427
+ "stateful_callbacks": {
1428
+ "TrainerControl": {
1429
+ "args": {
1430
+ "should_epoch_stop": false,
1431
+ "should_evaluate": false,
1432
+ "should_log": false,
1433
+ "should_save": true,
1434
+ "should_training_stop": true
1435
+ },
1436
+ "attributes": {}
1437
+ }
1438
+ },
1439
+ "total_flos": 0.0,
1440
+ "train_batch_size": 1,
1441
+ "trial_name": null,
1442
+ "trial_params": null
1443
+ }