[INFO|2025-05-29 19:37:34] configuration_utils.py:696 >> loading configuration file config.json from cache at /home/kiho/.cache/huggingface/hub/models--Qwen--Qwen2.5-Coder-7B-Instruct/snapshots/c03e6d358207e414f1eca0bb1891e29f1db0e242/config.json [INFO|2025-05-29 19:37:34] configuration_utils.py:768 >> Model config Qwen2Config { "_name_or_path": "Qwen/Qwen2.5-Coder-7B-Instruct", "architectures": [ "Qwen2ForCausalLM" ], "attention_dropout": 0.0, "bos_token_id": 151643, "eos_token_id": 151645, "hidden_act": "silu", "hidden_size": 3584, "initializer_range": 0.02, "intermediate_size": 18944, "max_position_embeddings": 32768, "max_window_layers": 28, "model_type": "qwen2", "num_attention_heads": 28, "num_hidden_layers": 28, "num_key_value_heads": 4, "rms_norm_eps": 1e-06, "rope_scaling": null, "rope_theta": 1000000.0, "sliding_window": null, "tie_word_embeddings": false, "torch_dtype": "bfloat16", "transformers_version": "4.48.2", "use_cache": true, "use_sliding_window": false, "vocab_size": 152064 } [INFO|2025-05-29 19:37:34] tokenization_utils_base.py:2034 >> loading file vocab.json from cache at /home/kiho/.cache/huggingface/hub/models--Qwen--Qwen2.5-Coder-7B-Instruct/snapshots/c03e6d358207e414f1eca0bb1891e29f1db0e242/vocab.json [INFO|2025-05-29 19:37:34] tokenization_utils_base.py:2034 >> loading file merges.txt from cache at /home/kiho/.cache/huggingface/hub/models--Qwen--Qwen2.5-Coder-7B-Instruct/snapshots/c03e6d358207e414f1eca0bb1891e29f1db0e242/merges.txt [INFO|2025-05-29 19:37:34] tokenization_utils_base.py:2034 >> loading file tokenizer.json from cache at /home/kiho/.cache/huggingface/hub/models--Qwen--Qwen2.5-Coder-7B-Instruct/snapshots/c03e6d358207e414f1eca0bb1891e29f1db0e242/tokenizer.json [INFO|2025-05-29 19:37:34] tokenization_utils_base.py:2034 >> loading file added_tokens.json from cache at None [INFO|2025-05-29 19:37:34] tokenization_utils_base.py:2034 >> loading file special_tokens_map.json from cache at None [INFO|2025-05-29 19:37:34] tokenization_utils_base.py:2034 >> loading file tokenizer_config.json from cache at /home/kiho/.cache/huggingface/hub/models--Qwen--Qwen2.5-Coder-7B-Instruct/snapshots/c03e6d358207e414f1eca0bb1891e29f1db0e242/tokenizer_config.json [INFO|2025-05-29 19:37:34] tokenization_utils_base.py:2034 >> loading file chat_template.jinja from cache at None [INFO|2025-05-29 19:37:34] tokenization_utils_base.py:2304 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. [INFO|2025-05-29 19:37:34] logging.py:157 >> Add <|im_end|> to stop words. [INFO|2025-05-29 19:37:34] logging.py:157 >> Loading dataset Codes_query_filtered_330k_ns_over8_1.json... [INFO|2025-05-29 19:37:47] configuration_utils.py:696 >> loading configuration file config.json from cache at /home/kiho/.cache/huggingface/hub/models--Qwen--Qwen2.5-Coder-7B-Instruct/snapshots/c03e6d358207e414f1eca0bb1891e29f1db0e242/config.json [INFO|2025-05-29 19:37:47] configuration_utils.py:768 >> Model config Qwen2Config { "_name_or_path": "Qwen/Qwen2.5-Coder-7B-Instruct", "architectures": [ "Qwen2ForCausalLM" ], "attention_dropout": 0.0, "bos_token_id": 151643, "eos_token_id": 151645, "hidden_act": "silu", "hidden_size": 3584, "initializer_range": 0.02, "intermediate_size": 18944, "max_position_embeddings": 32768, "max_window_layers": 28, "model_type": "qwen2", "num_attention_heads": 28, "num_hidden_layers": 28, "num_key_value_heads": 4, "rms_norm_eps": 1e-06, "rope_scaling": null, "rope_theta": 1000000.0, "sliding_window": null, "tie_word_embeddings": false, "torch_dtype": "bfloat16", "transformers_version": "4.48.2", "use_cache": true, "use_sliding_window": false, "vocab_size": 152064 } [WARNING|2025-05-29 19:37:47] logging.py:162 >> Input length is smaller than max length. Consider increase input length. [INFO|2025-05-29 19:37:47] logging.py:157 >> Using llama3 scaling strategy and setting scaling factor to 1.0. [INFO|2025-05-29 19:37:47] logging.py:157 >> Using block diagonal attention for sequence packing without cross-attention. [INFO|2025-05-29 19:37:47] logging.py:157 >> Liger kernel has been applied to the model. [INFO|2025-05-29 19:37:47] modeling_utils.py:3904 >> loading weights file model.safetensors from cache at /home/kiho/.cache/huggingface/hub/models--Qwen--Qwen2.5-Coder-7B-Instruct/snapshots/c03e6d358207e414f1eca0bb1891e29f1db0e242/model.safetensors.index.json [INFO|2025-05-29 19:37:47] modeling_utils.py:1582 >> Instantiating Qwen2ForCausalLM model under default dtype torch.bfloat16. [INFO|2025-05-29 19:37:47] configuration_utils.py:1140 >> Generate config GenerationConfig { "bos_token_id": 151643, "eos_token_id": 151645 } [INFO|2025-05-29 19:37:57] modeling_utils.py:4888 >> All model checkpoint weights were used when initializing Qwen2ForCausalLM. [INFO|2025-05-29 19:37:57] modeling_utils.py:4896 >> All the weights of Qwen2ForCausalLM were initialized from the model checkpoint at Qwen/Qwen2.5-Coder-7B-Instruct. If your task is similar to the task the model of the checkpoint was trained on, you can already use Qwen2ForCausalLM for predictions without further training. [INFO|2025-05-29 19:37:57] configuration_utils.py:1095 >> loading configuration file generation_config.json from cache at /home/kiho/.cache/huggingface/hub/models--Qwen--Qwen2.5-Coder-7B-Instruct/snapshots/c03e6d358207e414f1eca0bb1891e29f1db0e242/generation_config.json [INFO|2025-05-29 19:37:57] configuration_utils.py:1140 >> Generate config GenerationConfig { "bos_token_id": 151643, "do_sample": true, "eos_token_id": [ 151645, 151643 ], "pad_token_id": 151643, "repetition_penalty": 1.1, "temperature": 0.7, "top_k": 20, "top_p": 0.8 } [INFO|2025-05-29 19:37:57] logging.py:157 >> Gradient checkpointing enabled. [INFO|2025-05-29 19:37:57] logging.py:157 >> Using torch SDPA for faster training and inference. [INFO|2025-05-29 19:37:57] logging.py:157 >> Upcasting trainable params to float32. [INFO|2025-05-29 19:37:57] logging.py:157 >> Fine-tuning method: Freeze [INFO|2025-05-29 19:37:57] logging.py:157 >> Set trainable layers: .26.,.27. [INFO|2025-05-29 19:37:57] logging.py:157 >> trainable params: 466,115,584 || all params: 7,615,616,512 || trainable%: 6.1205 [INFO|2025-05-29 19:37:57] trainer.py:741 >> Using auto half precision backend [INFO|2025-05-29 19:37:57] logging.py:157 >> Found linear modules: k_proj,q_proj,o_proj,down_proj,v_proj,gate_proj,up_proj [INFO|2025-05-29 19:37:57] logging.py:157 >> Using APOLLO optimizer with args: {'rank': 256, 'proj': 'random', 'proj_type': 'std', 'update_proj_gap': 200, 'scale': 1, 'scale_type': 'channel', 'scale_front': False}. [INFO|2025-05-29 19:37:57] trainer.py:2369 >> ***** Running training ***** [INFO|2025-05-29 19:37:57] trainer.py:2370 >> Num examples = 4,739 [INFO|2025-05-29 19:37:57] trainer.py:2371 >> Num Epochs = 1 [INFO|2025-05-29 19:37:57] trainer.py:2372 >> Instantaneous batch size per device = 16 [INFO|2025-05-29 19:37:57] trainer.py:2375 >> Total train batch size (w. parallel, distributed & accumulation) = 512 [INFO|2025-05-29 19:37:57] trainer.py:2376 >> Gradient Accumulation steps = 8 [INFO|2025-05-29 19:37:57] trainer.py:2377 >> Total optimization steps = 9 [INFO|2025-05-29 19:37:57] trainer.py:2378 >> Number of trainable parameters = 466,115,584 [INFO|2025-05-29 19:39:46] logging.py:157 >> {'loss': 0.8250, 'learning_rate': 4.8492e-05, 'epoch': 0.11, 'throughput': 19409.05} [INFO|2025-05-29 19:41:28] logging.py:157 >> {'loss': 0.7867, 'learning_rate': 4.4151e-05, 'epoch': 0.21, 'throughput': 20051.40} [INFO|2025-05-29 19:43:10] logging.py:157 >> {'loss': 0.7849, 'learning_rate': 3.7500e-05, 'epoch': 0.32, 'throughput': 20211.57} [INFO|2025-05-29 19:44:51] logging.py:157 >> {'loss': 0.7609, 'learning_rate': 2.9341e-05, 'epoch': 0.43, 'throughput': 20309.39} [INFO|2025-05-29 19:46:33] logging.py:157 >> {'loss': 0.7475, 'learning_rate': 2.0659e-05, 'epoch': 0.53, 'throughput': 20377.09} [INFO|2025-05-29 19:48:15] logging.py:157 >> {'loss': 0.7783, 'learning_rate': 1.2500e-05, 'epoch': 0.64, 'throughput': 20417.33} [INFO|2025-05-29 19:49:56] logging.py:157 >> {'loss': 0.7453, 'learning_rate': 5.8489e-06, 'epoch': 0.75, 'throughput': 20443.35} [INFO|2025-05-29 19:51:38] logging.py:157 >> {'loss': 0.7393, 'learning_rate': 1.5077e-06, 'epoch': 0.85, 'throughput': 20461.76} [INFO|2025-05-29 19:53:20] logging.py:157 >> {'loss': 0.7599, 'learning_rate': 0.0000e+00, 'epoch': 0.96, 'throughput': 20477.77} [INFO|2025-05-29 19:53:20] trainer.py:3910 >> Saving model checkpoint to saves/Qwen2.5-Coder-7B-Instruct/freeze/qwen_nsx_8_1/checkpoint-9 [INFO|2025-05-29 19:53:20] configuration_utils.py:420 >> Configuration saved in saves/Qwen2.5-Coder-7B-Instruct/freeze/qwen_nsx_8_1/checkpoint-9/config.json [INFO|2025-05-29 19:53:20] configuration_utils.py:909 >> Configuration saved in saves/Qwen2.5-Coder-7B-Instruct/freeze/qwen_nsx_8_1/checkpoint-9/generation_config.json [INFO|2025-05-29 19:53:43] modeling_utils.py:2996 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at saves/Qwen2.5-Coder-7B-Instruct/freeze/qwen_nsx_8_1/checkpoint-9/model.safetensors.index.json. [INFO|2025-05-29 19:53:43] tokenization_utils_base.py:2491 >> tokenizer config file saved in saves/Qwen2.5-Coder-7B-Instruct/freeze/qwen_nsx_8_1/checkpoint-9/tokenizer_config.json [INFO|2025-05-29 19:53:43] tokenization_utils_base.py:2500 >> Special tokens file saved in saves/Qwen2.5-Coder-7B-Instruct/freeze/qwen_nsx_8_1/checkpoint-9/special_tokens_map.json [INFO|2025-05-29 19:53:43] trainer.py:2643 >> Training completed. Do not forget to share your model on huggingface.co/models =) [INFO|2025-05-29 19:53:43] trainer.py:3910 >> Saving model checkpoint to saves/Qwen2.5-Coder-7B-Instruct/freeze/qwen_nsx_8_1 [INFO|2025-05-29 19:53:43] configuration_utils.py:420 >> Configuration saved in saves/Qwen2.5-Coder-7B-Instruct/freeze/qwen_nsx_8_1/config.json [INFO|2025-05-29 19:53:43] configuration_utils.py:909 >> Configuration saved in saves/Qwen2.5-Coder-7B-Instruct/freeze/qwen_nsx_8_1/generation_config.json [INFO|2025-05-29 19:54:07] modeling_utils.py:2996 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 4 checkpoint shards. You can find where each parameters has been saved in the index located at saves/Qwen2.5-Coder-7B-Instruct/freeze/qwen_nsx_8_1/model.safetensors.index.json. [INFO|2025-05-29 19:54:07] tokenization_utils_base.py:2491 >> tokenizer config file saved in saves/Qwen2.5-Coder-7B-Instruct/freeze/qwen_nsx_8_1/tokenizer_config.json [INFO|2025-05-29 19:54:07] tokenization_utils_base.py:2500 >> Special tokens file saved in saves/Qwen2.5-Coder-7B-Instruct/freeze/qwen_nsx_8_1/special_tokens_map.json [WARNING|2025-05-29 19:54:07] logging.py:162 >> No metric eval_loss to plot. [WARNING|2025-05-29 19:54:07] logging.py:162 >> No metric eval_accuracy to plot. [INFO|2025-05-29 19:54:07] modelcard.py:449 >> Dropping the following result as it does not have all the necessary fields: {'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}}