Hugging Face's logo

PKU-Alignment
/
tinyllama-1T-IMDB-Q1-10000

PyTorch
llama
Model card Files
xet
Community
tinyllama-1T-IMDB-Q1-10000
File size: 49,240 Bytes 
c163c36
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
 
[2025-05-29 17:25:23,839] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Warning: The cache directory for DeepSpeed Triton autotune, /home/hansirui_1st/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
[2025-05-29 17:25:29,907] [WARNING] [runner.py:215:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2025-05-29 17:25:29,908] [INFO] [runner.py:605:main] cmd = /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/bin/python3.11 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgMywgNCwgNSwgNiwgN119 --master_addr=127.0.0.1 --master_port=51152 --module --enable_each_rank_log=None safe_rlhf.finetune --train_datasets inverse-json::/home/hansirui_1st/jiayi/resist/imdb_data/train/pos/10000/train.json --model_name_or_path /aifs4su/hansirui_1st/models/TinyLlama-1.1B-intermediate-step-480k-1T --max_length 512 --trust_remote_code True --epochs 1 --per_device_train_batch_size 1 --per_device_eval_batch_size 4 --gradient_accumulation_steps 8 --gradient_checkpointing --learning_rate 1e-5 --lr_warmup_ratio 0 --weight_decay 0.0 --lr_scheduler_type constant --weight_decay 0.0 --seed 42 --output_dir /aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-1T/tinyllama-1T-s3-Q1-10000 --log_type wandb --log_run_name imdb-tinyllama-1T-s3-Q1-10000 --log_project Inverse_Alignment_IMDb --zero_stage 3 --offload none --bf16 True --tf32 True --save_16bit
[2025-05-29 17:25:32,005] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Warning: The cache directory for DeepSpeed Triton autotune, /home/hansirui_1st/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
[2025-05-29 17:25:36,686] [INFO] [launch.py:146:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3, 4, 5, 6, 7]}
[2025-05-29 17:25:36,686] [INFO] [launch.py:152:main] nnodes=1, num_local_procs=8, node_rank=0
[2025-05-29 17:25:36,686] [INFO] [launch.py:163:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3, 4, 5, 6, 7]})
[2025-05-29 17:25:36,686] [INFO] [launch.py:164:main] dist_world_size=8
[2025-05-29 17:25:36,686] [INFO] [launch.py:168:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
[2025-05-29 17:25:36,687] [INFO] [launch.py:256:main] process 1990444 spawned with command: ['/aifs4su/hansirui_1st/miniconda3/envs/jy-resist/bin/python3.11', '-u', '-m', 'safe_rlhf.finetune', '--local_rank=0', '--train_datasets', 'inverse-json::/home/hansirui_1st/jiayi/resist/imdb_data/train/pos/10000/train.json', '--model_name_or_path', '/aifs4su/hansirui_1st/models/TinyLlama-1.1B-intermediate-step-480k-1T', '--max_length', '512', '--trust_remote_code', 'True', '--epochs', '1', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '4', '--gradient_accumulation_steps', '8', '--gradient_checkpointing', '--learning_rate', '1e-5', '--lr_warmup_ratio', '0', '--weight_decay', '0.0', '--lr_scheduler_type', 'constant', '--weight_decay', '0.0', '--seed', '42', '--output_dir', '/aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-1T/tinyllama-1T-s3-Q1-10000', '--log_type', 'wandb', '--log_run_name', 'imdb-tinyllama-1T-s3-Q1-10000', '--log_project', 'Inverse_Alignment_IMDb', '--zero_stage', '3', '--offload', 'none', '--bf16', 'True', '--tf32', 'True', '--save_16bit']
[2025-05-29 17:25:36,688] [INFO] [launch.py:256:main] process 1990445 spawned with command: ['/aifs4su/hansirui_1st/miniconda3/envs/jy-resist/bin/python3.11', '-u', '-m', 'safe_rlhf.finetune', '--local_rank=1', '--train_datasets', 'inverse-json::/home/hansirui_1st/jiayi/resist/imdb_data/train/pos/10000/train.json', '--model_name_or_path', '/aifs4su/hansirui_1st/models/TinyLlama-1.1B-intermediate-step-480k-1T', '--max_length', '512', '--trust_remote_code', 'True', '--epochs', '1', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '4', '--gradient_accumulation_steps', '8', '--gradient_checkpointing', '--learning_rate', '1e-5', '--lr_warmup_ratio', '0', '--weight_decay', '0.0', '--lr_scheduler_type', 'constant', '--weight_decay', '0.0', '--seed', '42', '--output_dir', '/aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-1T/tinyllama-1T-s3-Q1-10000', '--log_type', 'wandb', '--log_run_name', 'imdb-tinyllama-1T-s3-Q1-10000', '--log_project', 'Inverse_Alignment_IMDb', '--zero_stage', '3', '--offload', 'none', '--bf16', 'True', '--tf32', 'True', '--save_16bit']
[2025-05-29 17:25:36,688] [INFO] [launch.py:256:main] process 1990446 spawned with command: ['/aifs4su/hansirui_1st/miniconda3/envs/jy-resist/bin/python3.11', '-u', '-m', 'safe_rlhf.finetune', '--local_rank=2', '--train_datasets', 'inverse-json::/home/hansirui_1st/jiayi/resist/imdb_data/train/pos/10000/train.json', '--model_name_or_path', '/aifs4su/hansirui_1st/models/TinyLlama-1.1B-intermediate-step-480k-1T', '--max_length', '512', '--trust_remote_code', 'True', '--epochs', '1', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '4', '--gradient_accumulation_steps', '8', '--gradient_checkpointing', '--learning_rate', '1e-5', '--lr_warmup_ratio', '0', '--weight_decay', '0.0', '--lr_scheduler_type', 'constant', '--weight_decay', '0.0', '--seed', '42', '--output_dir', '/aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-1T/tinyllama-1T-s3-Q1-10000', '--log_type', 'wandb', '--log_run_name', 'imdb-tinyllama-1T-s3-Q1-10000', '--log_project', 'Inverse_Alignment_IMDb', '--zero_stage', '3', '--offload', 'none', '--bf16', 'True', '--tf32', 'True', '--save_16bit']
[2025-05-29 17:25:36,689] [INFO] [launch.py:256:main] process 1990447 spawned with command: ['/aifs4su/hansirui_1st/miniconda3/envs/jy-resist/bin/python3.11', '-u', '-m', 'safe_rlhf.finetune', '--local_rank=3', '--train_datasets', 'inverse-json::/home/hansirui_1st/jiayi/resist/imdb_data/train/pos/10000/train.json', '--model_name_or_path', '/aifs4su/hansirui_1st/models/TinyLlama-1.1B-intermediate-step-480k-1T', '--max_length', '512', '--trust_remote_code', 'True', '--epochs', '1', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '4', '--gradient_accumulation_steps', '8', '--gradient_checkpointing', '--learning_rate', '1e-5', '--lr_warmup_ratio', '0', '--weight_decay', '0.0', '--lr_scheduler_type', 'constant', '--weight_decay', '0.0', '--seed', '42', '--output_dir', '/aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-1T/tinyllama-1T-s3-Q1-10000', '--log_type', 'wandb', '--log_run_name', 'imdb-tinyllama-1T-s3-Q1-10000', '--log_project', 'Inverse_Alignment_IMDb', '--zero_stage', '3', '--offload', 'none', '--bf16', 'True', '--tf32', 'True', '--save_16bit']
[2025-05-29 17:25:36,689] [INFO] [launch.py:256:main] process 1990448 spawned with command: ['/aifs4su/hansirui_1st/miniconda3/envs/jy-resist/bin/python3.11', '-u', '-m', 'safe_rlhf.finetune', '--local_rank=4', '--train_datasets', 'inverse-json::/home/hansirui_1st/jiayi/resist/imdb_data/train/pos/10000/train.json', '--model_name_or_path', '/aifs4su/hansirui_1st/models/TinyLlama-1.1B-intermediate-step-480k-1T', '--max_length', '512', '--trust_remote_code', 'True', '--epochs', '1', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '4', '--gradient_accumulation_steps', '8', '--gradient_checkpointing', '--learning_rate', '1e-5', '--lr_warmup_ratio', '0', '--weight_decay', '0.0', '--lr_scheduler_type', 'constant', '--weight_decay', '0.0', '--seed', '42', '--output_dir', '/aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-1T/tinyllama-1T-s3-Q1-10000', '--log_type', 'wandb', '--log_run_name', 'imdb-tinyllama-1T-s3-Q1-10000', '--log_project', 'Inverse_Alignment_IMDb', '--zero_stage', '3', '--offload', 'none', '--bf16', 'True', '--tf32', 'True', '--save_16bit']
[2025-05-29 17:25:36,690] [INFO] [launch.py:256:main] process 1990449 spawned with command: ['/aifs4su/hansirui_1st/miniconda3/envs/jy-resist/bin/python3.11', '-u', '-m', 'safe_rlhf.finetune', '--local_rank=5', '--train_datasets', 'inverse-json::/home/hansirui_1st/jiayi/resist/imdb_data/train/pos/10000/train.json', '--model_name_or_path', '/aifs4su/hansirui_1st/models/TinyLlama-1.1B-intermediate-step-480k-1T', '--max_length', '512', '--trust_remote_code', 'True', '--epochs', '1', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '4', '--gradient_accumulation_steps', '8', '--gradient_checkpointing', '--learning_rate', '1e-5', '--lr_warmup_ratio', '0', '--weight_decay', '0.0', '--lr_scheduler_type', 'constant', '--weight_decay', '0.0', '--seed', '42', '--output_dir', '/aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-1T/tinyllama-1T-s3-Q1-10000', '--log_type', 'wandb', '--log_run_name', 'imdb-tinyllama-1T-s3-Q1-10000', '--log_project', 'Inverse_Alignment_IMDb', '--zero_stage', '3', '--offload', 'none', '--bf16', 'True', '--tf32', 'True', '--save_16bit']
[2025-05-29 17:25:36,690] [INFO] [launch.py:256:main] process 1990450 spawned with command: ['/aifs4su/hansirui_1st/miniconda3/envs/jy-resist/bin/python3.11', '-u', '-m', 'safe_rlhf.finetune', '--local_rank=6', '--train_datasets', 'inverse-json::/home/hansirui_1st/jiayi/resist/imdb_data/train/pos/10000/train.json', '--model_name_or_path', '/aifs4su/hansirui_1st/models/TinyLlama-1.1B-intermediate-step-480k-1T', '--max_length', '512', '--trust_remote_code', 'True', '--epochs', '1', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '4', '--gradient_accumulation_steps', '8', '--gradient_checkpointing', '--learning_rate', '1e-5', '--lr_warmup_ratio', '0', '--weight_decay', '0.0', '--lr_scheduler_type', 'constant', '--weight_decay', '0.0', '--seed', '42', '--output_dir', '/aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-1T/tinyllama-1T-s3-Q1-10000', '--log_type', 'wandb', '--log_run_name', 'imdb-tinyllama-1T-s3-Q1-10000', '--log_project', 'Inverse_Alignment_IMDb', '--zero_stage', '3', '--offload', 'none', '--bf16', 'True', '--tf32', 'True', '--save_16bit']
[2025-05-29 17:25:36,691] [INFO] [launch.py:256:main] process 1990451 spawned with command: ['/aifs4su/hansirui_1st/miniconda3/envs/jy-resist/bin/python3.11', '-u', '-m', 'safe_rlhf.finetune', '--local_rank=7', '--train_datasets', 'inverse-json::/home/hansirui_1st/jiayi/resist/imdb_data/train/pos/10000/train.json', '--model_name_or_path', '/aifs4su/hansirui_1st/models/TinyLlama-1.1B-intermediate-step-480k-1T', '--max_length', '512', '--trust_remote_code', 'True', '--epochs', '1', '--per_device_train_batch_size', '1', '--per_device_eval_batch_size', '4', '--gradient_accumulation_steps', '8', '--gradient_checkpointing', '--learning_rate', '1e-5', '--lr_warmup_ratio', '0', '--weight_decay', '0.0', '--lr_scheduler_type', 'constant', '--weight_decay', '0.0', '--seed', '42', '--output_dir', '/aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-1T/tinyllama-1T-s3-Q1-10000', '--log_type', 'wandb', '--log_run_name', 'imdb-tinyllama-1T-s3-Q1-10000', '--log_project', 'Inverse_Alignment_IMDb', '--zero_stage', '3', '--offload', 'none', '--bf16', 'True', '--tf32', 'True', '--save_16bit']
[2025-05-29 17:25:42,632] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-05-29 17:25:42,684] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-05-29 17:25:42,718] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-05-29 17:25:42,782] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-05-29 17:25:42,793] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-05-29 17:25:42,800] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-05-29 17:25:42,809] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Warning: The cache directory for DeepSpeed Triton autotune, /home/hansirui_1st/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
[2025-05-29 17:25:42,826] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Warning: The cache directory for DeepSpeed Triton autotune, /home/hansirui_1st/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /home/hansirui_1st/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /home/hansirui_1st/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /home/hansirui_1st/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /home/hansirui_1st/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /home/hansirui_1st/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /home/hansirui_1st/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
[2025-05-29 17:25:50,079] [INFO] [comm.py:669:init_distributed] cdb=None
[2025-05-29 17:25:50,079] [INFO] [comm.py:669:init_distributed] cdb=None
[2025-05-29 17:25:50,079] [INFO] [comm.py:669:init_distributed] cdb=None
[2025-05-29 17:25:50,206] [INFO] [comm.py:669:init_distributed] cdb=None
[2025-05-29 17:25:50,206] [INFO] [comm.py:669:init_distributed] cdb=None
[2025-05-29 17:25:50,206] [INFO] [comm.py:669:init_distributed] cdb=None
[2025-05-29 17:25:50,206] [INFO] [comm.py:669:init_distributed] cdb=None
[2025-05-29 17:25:50,206] [INFO] [comm.py:700:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2025-05-29 17:25:50,209] [INFO] [comm.py:669:init_distributed] cdb=None
Set logger level to INFO.
[2025-05-29 17:26:00,590] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
[2025-05-29 17:26:00,590] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
[2025-05-29 17:26:00,590] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
[2025-05-29 17:26:00,590] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
[2025-05-29 17:26:00,590] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
[2025-05-29 17:26:00,590] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
[2025-05-29 17:26:00,590] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
[2025-05-29 17:26:00,590] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
[2025-05-29 17:26:00,719] [INFO] [partition_parameters.py:348:__exit__] finished initializing model - num_params = 201, num_elems = 1.10B
[1/3] /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/bin/nvcc --generate-dependencies-with-compile --dependency-output multi_tensor_adam.cuda.o.d -ccbin /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/bin/x86_64-conda-linux-gnu-cc -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -I/aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib/python3.11/site-packages/deepspeed/ops/csrc/adam -isystem /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib/python3.11/site-packages/torch/include -isystem /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib/python3.11/site-packages/torch/include/TH -isystem /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib/python3.11/site-packages/torch/include/THC -isystem /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/include -isystem /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_90,code=compute_90 -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -lineinfo --use_fast_math -gencode=arch=compute_90,code=sm_90 -gencode=arch=compute_90,code=compute_90 -DBF16_AVAILABLE -U__CUDA_NO_BFLOAT16_OPERATORS__ -U__CUDA_NO_BFLOAT162_OPERATORS__ -U__CUDA_NO_BFLOAT16_CONVERSIONS__ -std=c++17 -c /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib/python3.11/site-packages/deepspeed/ops/csrc/adam/multi_tensor_adam.cu -o multi_tensor_adam.cuda.o 
nvcc warning : incompatible redefinition for option 'compiler-bindir', the last value of this option was used
[2/3] /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/bin/x86_64-conda-linux-gnu-c++ -MMD -MF fused_adam_frontend.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -I/aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib/python3.11/site-packages/deepspeed/ops/csrc/adam -isystem /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib/python3.11/site-packages/torch/include -isystem /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib/python3.11/site-packages/torch/include/TH -isystem /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib/python3.11/site-packages/torch/include/THC -isystem /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/include -isystem /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DBF16_AVAILABLE -c /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib/python3.11/site-packages/deepspeed/ops/csrc/adam/fused_adam_frontend.cpp -o fused_adam_frontend.o 
[3/3] /aifs4su/hansirui_1st/miniconda3/envs/jy-resist/bin/x86_64-conda-linux-gnu-c++ fused_adam_frontend.o multi_tensor_adam.cuda.o -shared -L/aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib/python3.11/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/aifs4su/hansirui_1st/miniconda3/envs/jy-resist/lib -lcudart -o fused_adam.so
Time to load fused_adam op: 24.022887468338013 seconds
[2025-05-29 17:26:26,884] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
Time to load fused_adam op: 24.023088693618774 secondsTime to load fused_adam op: 24.023180723190308 seconds

Time to load fused_adam op: 24.023329257965088 seconds
Time to load fused_adam op: 24.022546768188477 seconds
Time to load fused_adam op: 24.023573875427246 seconds
[2025-05-29 17:26:26,884] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed info: version=0.16.8, git-hash=unknown, git-branch=unknown
[2025-05-29 17:26:26,884] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
[2025-05-29 17:26:26,885] [INFO] [comm.py:694:init_distributed] Distributed backend already initialized
[2025-05-29 17:26:26,885] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
[2025-05-29 17:26:26,885] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
[2025-05-29 17:26:26,885] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
Time to load fused_adam op: 24.025002479553223 seconds
[2025-05-29 17:26:26,885] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
[2025-05-29 17:26:26,886] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
Time to load fused_adam op: 24.025075435638428 seconds
[2025-05-29 17:26:26,886] [INFO] [config.py:735:__init__] Config mesh_device None world_size = 8
[2025-05-29 17:26:26,889] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2025-05-29 17:26:26,890] [INFO] [logging.py:107:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2025-05-29 17:26:26,890] [INFO] [logging.py:107:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2025-05-29 17:26:26,892] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam
[2025-05-29 17:26:26,892] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=FusedAdam type=<class 'deepspeed.ops.adam.fused_adam.FusedAdam'>
[2025-05-29 17:26:26,892] [INFO] [logging.py:107:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False
[2025-05-29 17:26:26,892] [INFO] [logging.py:107:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer
[2025-05-29 17:26:27,157] [INFO] [utils.py:781:see_memory_usage] Stage 3 initialize beginning
[2025-05-29 17:26:27,158] [INFO] [utils.py:782:see_memory_usage] MA 0.5 GB         Max_MA 1.24 GB         CA 1.52 GB         Max_CA 2 GB 
[2025-05-29 17:26:27,158] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 100.18 GB, percent = 5.0%
[2025-05-29 17:26:27,159] [INFO] [stage3.py:170:__init__] Reduce bucket size 500000000
[2025-05-29 17:26:27,159] [INFO] [stage3.py:171:__init__] Prefetch bucket size 30000000
[2025-05-29 17:26:27,301] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [begin]
[2025-05-29 17:26:27,301] [INFO] [utils.py:782:see_memory_usage] MA 0.5 GB         Max_MA 0.5 GB         CA 1.52 GB         Max_CA 2 GB 
[2025-05-29 17:26:27,301] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 100.2 GB, percent = 5.0%
Parameter Offload: Total persistent parameters: 92160 in 45 params
[2025-05-29 17:26:27,430] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [end]
[2025-05-29 17:26:27,431] [INFO] [utils.py:782:see_memory_usage] MA 0.29 GB         Max_MA 0.52 GB         CA 1.52 GB         Max_CA 2 GB 
[2025-05-29 17:26:27,431] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 100.2 GB, percent = 5.0%
[2025-05-29 17:26:27,536] [INFO] [utils.py:781:see_memory_usage] Before creating fp16 partitions
[2025-05-29 17:26:27,537] [INFO] [utils.py:782:see_memory_usage] MA 0.29 GB         Max_MA 0.29 GB         CA 1.52 GB         Max_CA 2 GB 
[2025-05-29 17:26:27,537] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 100.2 GB, percent = 5.0%
[2025-05-29 17:26:28,129] [INFO] [utils.py:781:see_memory_usage] After creating fp16 partitions: 2
[2025-05-29 17:26:28,129] [INFO] [utils.py:782:see_memory_usage] MA 0.29 GB         Max_MA 0.29 GB         CA 0.5 GB         Max_CA 2 GB 
[2025-05-29 17:26:28,129] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 101.38 GB, percent = 5.0%
[2025-05-29 17:26:28,236] [INFO] [utils.py:781:see_memory_usage] Before creating fp32 partitions
[2025-05-29 17:26:28,236] [INFO] [utils.py:782:see_memory_usage] MA 0.29 GB         Max_MA 0.29 GB         CA 0.5 GB         Max_CA 1 GB 
[2025-05-29 17:26:28,236] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 101.38 GB, percent = 5.0%
[2025-05-29 17:26:28,343] [INFO] [utils.py:781:see_memory_usage] After creating fp32 partitions
[2025-05-29 17:26:28,344] [INFO] [utils.py:782:see_memory_usage] MA 0.8 GB         Max_MA 1.06 GB         CA 1.28 GB         Max_CA 1 GB 
[2025-05-29 17:26:28,344] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 101.38 GB, percent = 5.0%
[2025-05-29 17:26:28,451] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states
[2025-05-29 17:26:28,451] [INFO] [utils.py:782:see_memory_usage] MA 0.8 GB         Max_MA 0.8 GB         CA 1.28 GB         Max_CA 1 GB 
[2025-05-29 17:26:28,451] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 101.38 GB, percent = 5.0%
[2025-05-29 17:26:28,565] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states
[2025-05-29 17:26:28,565] [INFO] [utils.py:782:see_memory_usage] MA 0.8 GB         Max_MA 1.31 GB         CA 1.79 GB         Max_CA 2 GB 
[2025-05-29 17:26:28,566] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 101.38 GB, percent = 5.0%
[2025-05-29 17:26:28,566] [INFO] [stage3.py:534:_setup_for_real_optimizer] optimizer state initialized
[2025-05-29 17:26:28,722] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer
[2025-05-29 17:26:28,722] [INFO] [utils.py:782:see_memory_usage] MA 1.99 GB         Max_MA 2.23 GB         CA 2.72 GB         Max_CA 3 GB 
[2025-05-29 17:26:28,722] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 101.38 GB, percent = 5.0%
[2025-05-29 17:26:28,722] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer_Stage3
[2025-05-29 17:26:28,722] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed using client LR scheduler
[2025-05-29 17:26:28,722] [INFO] [logging.py:107:log_dist] [Rank 0] DeepSpeed LR Scheduler = <torch.optim.lr_scheduler.LambdaLR object at 0x1550cc0eb3d0>
[2025-05-29 17:26:28,723] [INFO] [logging.py:107:log_dist] [Rank 0] step=0, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:26:28,723] [INFO] [config.py:1003:print] DeepSpeedEngine configuration:
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'intra_op_parallelism': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   amp_enabled .................. False
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   amp_params ................... False
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   bfloat16_enabled ............. True
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   bfloat16_immediate_grad_update  True
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   checkpoint_parallel_write_pipeline  False
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   checkpoint_tag_validation_enabled  True
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   checkpoint_tag_validation_fail  False
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x1550cc0eca50>
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   communication_data_type ...... None
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   compile_config ............... deepcompile=False free_activation=False offload_activation=False offload_opt_states=False double_buffer=True symmetric_memory=False debug_log=False offload_parameters=False sync_before_reduce=False sync_after_reduce=False sync_before_allgather=False sync_after_allgather=False
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   curriculum_enabled_legacy .... False
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   curriculum_params_legacy ..... False
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'pin_memory': False, 'curriculum_learning': {'enabled': False}, 'dynamic_batching': {'enabled': False, 'lr_scaling_method': 'linear', 'min_batch_size': 1, 'max_batch_size': None, 'sequence_picking_order': 'dataloader', 'verbose': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   data_efficiency_enabled ...... False
[2025-05-29 17:26:28,723] [INFO] [config.py:1007:print]   dataloader_drop_last ......... False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   disable_allgather ............ False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   dump_state ................... False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   dynamic_loss_scale_args ...... None
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   eigenvalue_enabled ........... False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   eigenvalue_gas_boundary_resolution  1
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   eigenvalue_layer_num ......... 0
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   eigenvalue_max_iter .......... 100
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   eigenvalue_stability ......... 1e-06
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   eigenvalue_tol ............... 0.01
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   eigenvalue_verbose ........... False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   elasticity_enabled ........... False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   flops_profiler_config ........ {
    "enabled": false, 
    "recompute_fwd_factor": 0.0, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   fp16_auto_cast ............... None
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   fp16_enabled ................. False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   fp16_master_weights_and_gradients  False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   global_rank .................. 0
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   grad_accum_dtype ............. None
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   gradient_accumulation_steps .. 8
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   gradient_clipping ............ 1.0
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   gradient_predivide_factor .... 1.0
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   graph_harvesting ............. False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   initial_dynamic_scale ........ 1
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   load_universal_checkpoint .... False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   loss_scale ................... 1.0
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   memory_breakdown ............. False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   mics_hierarchial_params_gather  False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   mics_shard_size .............. -1
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   optimizer_legacy_fusion ...... False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   optimizer_name ............... None
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   optimizer_params ............. None
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   pld_enabled .................. False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   pld_params ................... False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   prescale_gradients ........... False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   scheduler_name ............... None
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   scheduler_params ............. None
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   seq_parallel_communication_data_type  torch.float32
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   sparse_attention ............. None
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   sparse_gradients_enabled ..... False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   steps_per_print .............. 10
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   tensor_parallel_config ....... dtype=torch.float16 autotp_size=0 tp_overlap_comm=False tensor_parallel=TPConfig(tp_size=1, tp_grain_size=1, mpu=None, tp_group=None) injection_policy_tuple=None keep_module_on_host=False replace_with_kernel_inject=False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   timers_config ................ enabled=True synchronized=True
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   train_batch_size ............. 64
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   train_micro_batch_size_per_gpu  1
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   use_data_before_expert_parallel_  False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   use_node_local_storage ....... False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   wall_clock_breakdown ......... False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   weight_quantization_config ... None
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   world_size ................... 8
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   zero_allow_untested_optimizer  False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   zero_config .................. stage=3 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='none', nvme_path=None, buffer_count=5, buffer_size=100000000, max_in_cpu=1000000000, pin_memory=False) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='none', nvme_path=None, buffer_count=4, pin_memory=False, pipeline_read=False, pipeline_write=False, fast_init=False, ratio=1.0) sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=30000000 param_persistence_threshold=10000 model_persistence_threshold=9223372036854775807 max_live_parameters=30000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=True module_granularity_threshold=0 use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False zeropp_loco_param=None mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=False pipeline_loading_checkpoint=False override_module_apply=True log_trace_cache_warnings=False
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   zero_enabled ................. True
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   zero_force_ds_cpu_optimizer .. True
[2025-05-29 17:26:28,724] [INFO] [config.py:1007:print]   zero_optimization_stage ...... 3
[2025-05-29 17:26:28,724] [INFO] [config.py:993:print_user_config]   json = {
    "train_batch_size": 64, 
    "train_micro_batch_size_per_gpu": 1, 
    "gradient_accumulation_steps": 8, 
    "steps_per_print": 10, 
    "zero_optimization": {
        "stage": 3, 
        "offload_param": {
            "device": "none"
        }, 
        "offload_optimizer": {
            "device": "none"
        }, 
        "param_persistence_threshold": 1.000000e+04, 
        "max_live_parameters": 3.000000e+07, 
        "prefetch_bucket_size": 3.000000e+07, 
        "memory_efficient_linear": false, 
        "gather_16bit_weights_on_model_save": true
    }, 
    "gradient_clipping": 1.0, 
    "prescale_gradients": false, 
    "wall_clock_breakdown": false, 
    "hybrid_engine": {
        "enabled": false, 
        "max_out_tokens": 512, 
        "inference_tp_size": 1, 
        "release_inference_cache": false, 
        "pin_parameters": true, 
        "tp_gather_partition_size": 8
    }, 
    "bf16": {
        "enabled": true
    }
}
***** Running training *****
[2025-05-29 17:27:04,893] [INFO] [logging.py:107:log_dist] [Rank 0] step=10, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:27:04,893] [INFO] [timer.py:264:stop] epoch=0/micro_step=80/global_step=10, RunningAvgSamplesPerSec=24.808672249043582, CurrSamplesPerSec=25.297381782600123, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
[2025-05-29 17:27:33,185] [INFO] [logging.py:107:log_dist] [Rank 0] step=20, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:27:33,185] [INFO] [timer.py:264:stop] epoch=0/micro_step=160/global_step=20, RunningAvgSamplesPerSec=24.802282095788485, CurrSamplesPerSec=26.76412254630623, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
[2025-05-29 17:28:01,875] [INFO] [logging.py:107:log_dist] [Rank 0] step=30, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:28:01,875] [INFO] [timer.py:264:stop] epoch=0/micro_step=240/global_step=30, RunningAvgSamplesPerSec=24.657336175411018, CurrSamplesPerSec=18.60149973048394, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
[2025-05-29 17:28:29,934] [INFO] [logging.py:107:log_dist] [Rank 0] step=40, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:28:29,935] [INFO] [timer.py:264:stop] epoch=0/micro_step=320/global_step=40, RunningAvgSamplesPerSec=24.763864887504017, CurrSamplesPerSec=23.82686189304851, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
[2025-05-29 17:28:57,730] [INFO] [logging.py:107:log_dist] [Rank 0] step=50, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:28:57,731] [INFO] [timer.py:264:stop] epoch=0/micro_step=400/global_step=50, RunningAvgSamplesPerSec=24.86547859489489, CurrSamplesPerSec=24.407891405542394, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
[2025-05-29 17:29:26,045] [INFO] [logging.py:107:log_dist] [Rank 0] step=60, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:29:26,046] [INFO] [timer.py:264:stop] epoch=0/micro_step=480/global_step=60, RunningAvgSamplesPerSec=24.84555662268532, CurrSamplesPerSec=19.488229452551018, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
[2025-05-29 17:29:56,772] [INFO] [logging.py:107:log_dist] [Rank 0] step=70, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:29:56,773] [INFO] [timer.py:264:stop] epoch=0/micro_step=560/global_step=70, RunningAvgSamplesPerSec=24.559091585937676, CurrSamplesPerSec=24.19412838995136, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
[2025-05-29 17:30:25,188] [INFO] [logging.py:107:log_dist] [Rank 0] step=80, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:30:25,189] [INFO] [timer.py:264:stop] epoch=0/micro_step=640/global_step=80, RunningAvgSamplesPerSec=24.581021958689888, CurrSamplesPerSec=23.995545082163574, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
[2025-05-29 17:30:52,972] [INFO] [logging.py:107:log_dist] [Rank 0] step=90, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:30:52,973] [INFO] [timer.py:264:stop] epoch=0/micro_step=720/global_step=90, RunningAvgSamplesPerSec=24.6615862281204, CurrSamplesPerSec=25.73261411037068, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
[2025-05-29 17:31:21,814] [INFO] [logging.py:107:log_dist] [Rank 0] step=100, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:31:21,814] [INFO] [timer.py:264:stop] epoch=0/micro_step=800/global_step=100, RunningAvgSamplesPerSec=24.63429072595949, CurrSamplesPerSec=23.93177130648976, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
[2025-05-29 17:31:50,359] [INFO] [logging.py:107:log_dist] [Rank 0] step=110, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:31:50,360] [INFO] [timer.py:264:stop] epoch=0/micro_step=880/global_step=110, RunningAvgSamplesPerSec=24.62462545662672, CurrSamplesPerSec=23.81174347363934, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
[2025-05-29 17:32:19,222] [INFO] [logging.py:107:log_dist] [Rank 0] step=120, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:32:19,223] [INFO] [timer.py:264:stop] epoch=0/micro_step=960/global_step=120, RunningAvgSamplesPerSec=24.588312839504304, CurrSamplesPerSec=24.2087189927535, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
[2025-05-29 17:32:48,245] [INFO] [logging.py:107:log_dist] [Rank 0] step=130, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:32:48,246] [INFO] [timer.py:264:stop] epoch=0/micro_step=1040/global_step=130, RunningAvgSamplesPerSec=24.564578322402998, CurrSamplesPerSec=25.180931313187962, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
[2025-05-29 17:33:16,874] [INFO] [logging.py:107:log_dist] [Rank 0] step=140, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:33:16,874] [INFO] [timer.py:264:stop] epoch=0/micro_step=1120/global_step=140, RunningAvgSamplesPerSec=24.550158022424334, CurrSamplesPerSec=24.569923065837752, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
[2025-05-29 17:33:45,067] [INFO] [logging.py:107:log_dist] [Rank 0] step=150, skipped=0, lr=[1e-05, 1e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2025-05-29 17:33:45,067] [INFO] [timer.py:264:stop] epoch=0/micro_step=1200/global_step=150, RunningAvgSamplesPerSec=24.570013808126674, CurrSamplesPerSec=25.659681796412837, MemAllocated=3.04GB, MaxMemAllocated=4.24GB
Saving model to "/aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-1T/tinyllama-1T-s3-Q1-10000" ...
Saving 16-bit model...
[2025-05-29 17:34:04,626] [INFO] [logging.py:107:log_dist] [Rank 0] [Torch] Checkpoint global_step156 is about to be saved!
[2025-05-29 17:34:04,626] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step156 is ready now!
[2025-05-29 17:34:04,626] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step156 is ready now!
[2025-05-29 17:34:04,626] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step156 is ready now!
[2025-05-29 17:34:04,626] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step156 is ready now!
[2025-05-29 17:34:04,626] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step156 is ready now!
[2025-05-29 17:34:04,627] [INFO] [engine.py:3860:save_16bit_model] Saving model weights to /aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-1T/tinyllama-1T-s3-Q1-10000/pytorch_model.bin, tag: global_step156
[2025-05-29 17:34:04,627] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step156 is ready now!
[2025-05-29 17:34:04,627] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step156 is ready now!
[2025-05-29 17:34:04,627] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving /aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-1T/tinyllama-1T-s3-Q1-10000/pytorch_model.bin...
[2025-05-29 17:34:06,599] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved /aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-1T/tinyllama-1T-s3-Q1-10000/pytorch_model.bin.
[2025-05-29 17:34:06,600] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step156 is ready now!
Model saved!
[2025-05-29 17:34:06,746] [INFO] [launch.py:351:main] Process 1990450 exits successfully.
[2025-05-29 17:34:06,746] [INFO] [launch.py:351:main] Process 1990446 exits successfully.
[2025-05-29 17:34:07,747] [INFO] [launch.py:351:main] Process 1990449 exits successfully.
[2025-05-29 17:34:07,747] [INFO] [launch.py:351:main] Process 1990448 exits successfully.
[2025-05-29 17:34:07,747] [INFO] [launch.py:351:main] Process 1990447 exits successfully.
[2025-05-29 17:34:07,747] [INFO] [launch.py:351:main] Process 1990451 exits successfully.
[2025-05-29 17:34:07,747] [INFO] [launch.py:351:main] Process 1990445 exits successfully.
wandb: 
wandb: 🚀 View run imdb-tinyllama-1T-s3-Q1-10000 at: https://wandb.ai/xtom/Inverse_Alignment_IMDb/runs/0t1vq5hu
wandb: Find logs at: ../../../../../../aifs4su/hansirui_1st/jiayi/setting3-imdb/tinyllama-1T/tinyllama-1T-s3-Q1-10000/wandb/run-20250529_172629-0t1vq5hu/logs
[2025-05-29 17:34:09,747] [INFO] [launch.py:351:main] Process 1990444 exits successfully.