2024-11-13 17:10:07,434 INFO StreamThr :1939 [internal.py:wandb_internal():86] W&B internal server running at pid: 1939, started at: 2024-11-13 17:10:07.433254 2024-11-13 17:10:07,435 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status 2024-11-13 17:10:07,436 INFO WriterThread:1939 [datastore.py:open_for_write():87] open: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/run-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1.wandb 2024-11-13 17:10:07,437 DEBUG SenderThread:1939 [sender.py:send():382] send: header 2024-11-13 17:10:07,443 DEBUG SenderThread:1939 [sender.py:send():382] send: run 2024-11-13 17:10:07,751 INFO SenderThread:1939 [dir_watcher.py:__init__():211] watching files in: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files 2024-11-13 17:10:07,751 INFO SenderThread:1939 [sender.py:_start_run_threads():1136] run started: kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1 with start time 1731517807.43302 2024-11-13 17:10:07,759 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: check_version 2024-11-13 17:10:07,760 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: check_version 2024-11-13 17:10:07,839 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: run_start 2024-11-13 17:10:07,929 DEBUG HandlerThread:1939 [system_info.py:__init__():27] System info init 2024-11-13 17:10:07,929 DEBUG HandlerThread:1939 [system_info.py:__init__():42] System info init done 2024-11-13 17:10:07,929 INFO HandlerThread:1939 [system_monitor.py:start():194] Starting system monitor 2024-11-13 17:10:07,929 INFO SystemMonitor:1939 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-11-13 17:10:07,930 INFO HandlerThread:1939 [system_monitor.py:probe():214] Collecting system info 2024-11-13 17:10:07,930 INFO SystemMonitor:1939 [interfaces.py:start():190] Started cpu monitoring 2024-11-13 17:10:07,931 INFO SystemMonitor:1939 [interfaces.py:start():190] Started disk monitoring 2024-11-13 17:10:07,932 INFO SystemMonitor:1939 [interfaces.py:start():190] Started gpu monitoring 2024-11-13 17:10:07,933 INFO SystemMonitor:1939 [interfaces.py:start():190] Started memory monitoring 2024-11-13 17:10:07,933 INFO SystemMonitor:1939 [interfaces.py:start():190] Started network monitoring 2024-11-13 17:10:07,993 DEBUG HandlerThread:1939 [system_info.py:probe():151] Probing system 2024-11-13 17:10:07,996 DEBUG HandlerThread:1939 [system_info.py:_probe_git():136] Probing git 2024-11-13 17:10:08,005 DEBUG HandlerThread:1939 [system_info.py:_probe_git():144] Probing git done 2024-11-13 17:10:08,005 DEBUG HandlerThread:1939 [system_info.py:probe():199] Probing system done 2024-11-13 17:10:08,005 DEBUG HandlerThread:1939 [system_monitor.py:probe():223] {'os': 'Linux-5.10.226-214.879.amzn2.x86_64-x86_64-with-glibc2.31', 'python': '3.10.12', 'heartbeatAt': '2024-11-13T17:10:07.993791', 'startedAt': '2024-11-13T17:10:07.428221', 'docker': None, 'cuda': None, 'args': ('--batch_size', '64', '--cache_dir', '/opt/ml/data/input/.cache', '--dataset', 'code-contests', '--gradient_accumulation_steps', '8', '--logging_steps', '5', '--lr', '1e-5', '--max_seq_length', '2048', '--model_name', 'Qwen/Qwen2.5-Coder-7B-Instruct', '--num_train_epochs', '2', '--output_dir', '/opt/ml/model/', '--push_to_hub', 'True', '--run_name', 'sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048', '--save_steps', '100', '--schedule', 'cosine', '--wandb_project', 'sft-codecontests-1112', '--weight_decay', '0.0'), 'state': 'running', 'program': '/opt/ml/code/qlearning_reasoning/training/sft.py', 'codePathLocal': 'qlearning_reasoning/training/sft.py', 'codePath': 'qlearning_reasoning/training/sft.py', 'git': {'remote': 'git@github.com:TRI-ML/reasoning-value-verifiers.git', 'commit': 'cbfd05147f15c55be83f4996eebdb305c17f119d'}, 'email': None, 'root': '/opt/ml/code', 'host': 'algo-1', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 96, 'cpu_count_logical': 192, 'cpu_freq': {'current': 2807.4830677083337, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3476.342, 'min': 0.0, 'max': 0.0}, {'current': 3247.882, 'min': 0.0, 'max': 0.0}, {'current': 2904.031, 'min': 0.0, 'max': 0.0}, {'current': 3109.433, 'min': 0.0, 'max': 0.0}, {'current': 3599.377, 'min': 0.0, 'max': 0.0}, {'current': 3103.902, 'min': 0.0, 'max': 0.0}, {'current': 3160.704, 'min': 0.0, 'max': 0.0}, {'current': 2903.061, 'min': 0.0, 'max': 0.0}, {'current': 3151.516, 'min': 0.0, 'max': 0.0}, {'current': 3220.914, 'min': 0.0, 'max': 0.0}, {'current': 3113.9, 'min': 0.0, 'max': 0.0}, {'current': 3599.288, 'min': 0.0, 'max': 0.0}, {'current': 3149.924, 'min': 0.0, 'max': 0.0}, {'current': 3195.68, 'min': 0.0, 'max': 0.0}, {'current': 3273.352, 'min': 0.0, 'max': 0.0}, {'current': 3152.916, 'min': 0.0, 'max': 0.0}, {'current': 2940.821, 'min': 0.0, 'max': 0.0}, {'current': 2997.447, 'min': 0.0, 'max': 0.0}, {'current': 2722.682, 'min': 0.0, 'max': 0.0}, {'current': 2960.205, 'min': 0.0, 'max': 0.0}, {'current': 3014.718, 'min': 0.0, 'max': 0.0}, {'current': 2865.342, 'min': 0.0, 'max': 0.0}, {'current': 2643.892, 'min': 0.0, 'max': 0.0}, {'current': 2626.179, 'min': 0.0, 'max': 0.0}, {'current': 3181.377, 'min': 0.0, 'max': 0.0}, {'current': 3271.231, 'min': 0.0, 'max': 0.0}, {'current': 3187.187, 'min': 0.0, 'max': 0.0}, {'current': 3199.812, 'min': 0.0, 'max': 0.0}, {'current': 3207.165, 'min': 0.0, 'max': 0.0}, {'current': 3251.62, 'min': 0.0, 'max': 0.0}, {'current': 3597.089, 'min': 0.0, 'max': 0.0}, {'current': 3222.446, 'min': 0.0, 'max': 0.0}, {'current': 3597.008, 'min': 0.0, 'max': 0.0}, {'current': 3199.923, 'min': 0.0, 'max': 0.0}, {'current': 3328.709, 'min': 0.0, 'max': 0.0}, {'current': 3189.821, 'min': 0.0, 'max': 0.0}, {'current': 3176.968, 'min': 0.0, 'max': 0.0}, {'current': 3170.487, 'min': 0.0, 'max': 0.0}, {'current': 3186.086, 'min': 0.0, 'max': 0.0}, {'current': 3216.688, 'min': 0.0, 'max': 0.0}, {'current': 3594.347, 'min': 0.0, 'max': 0.0}, {'current': 3039.479, 'min': 0.0, 'max': 0.0}, {'current': 3486.888, 'min': 0.0, 'max': 0.0}, {'current': 3343.804, 'min': 0.0, 'max': 0.0}, {'current': 3238.757, 'min': 0.0, 'max': 0.0}, {'current': 2966.737, 'min': 0.0, 'max': 0.0}, {'current': 3348.466, 'min': 0.0, 'max': 0.0}, {'current': 2907.859, 'min': 0.0, 'max': 0.0}, {'current': 3002.238, 'min': 0.0, 'max': 0.0}, {'current': 2217.1, 'min': 0.0, 'max': 0.0}, {'current': 2241.607, 'min': 0.0, 'max': 0.0}, {'current': 2684.27, 'min': 0.0, 'max': 0.0}, {'current': 2719.247, 'min': 0.0, 'max': 0.0}, {'current': 2783.971, 'min': 0.0, 'max': 0.0}, {'current': 2773.016, 'min': 0.0, 'max': 0.0}, {'current': 3086.799, 'min': 0.0, 'max': 0.0}, {'current': 2611.966, 'min': 0.0, 'max': 0.0}, {'current': 2540.393, 'min': 0.0, 'max': 0.0}, {'current': 2541.301, 'min': 0.0, 'max': 0.0}, {'current': 2543.134, 'min': 0.0, 'max': 0.0}, {'current': 1889.373, 'min': 0.0, 'max': 0.0}, {'current': 2536.698, 'min': 0.0, 'max': 0.0}, {'current': 1960.209, 'min': 0.0, 'max': 0.0}, {'current': 1871.901, 'min': 0.0, 'max': 0.0}, {'current': 2935.658, 'min': 0.0, 'max': 0.0}, {'current': 2920.588, 'min': 0.0, 'max': 0.0}, {'current': 3597.066, 'min': 0.0, 'max': 0.0}, {'current': 3324.97, 'min': 0.0, 'max': 0.0}, {'current': 3298.432, 'min': 0.0, 'max': 0.0}, {'current': 3404.565, 'min': 0.0, 'max': 0.0}, {'current': 2925.92, 'min': 0.0, 'max': 0.0}, {'current': 3278.833, 'min': 0.0, 'max': 0.0}, {'current': 2607.119, 'min': 0.0, 'max': 0.0}, {'current': 1884.144, 'min': 0.0, 'max': 0.0}, {'current': 2516.733, 'min': 0.0, 'max': 0.0}, {'current': 1885.068, 'min': 0.0, 'max': 0.0}, {'current': 1881.651, 'min': 0.0, 'max': 0.0}, {'current': 2500.279, 'min': 0.0, 'max': 0.0}, {'current': 2519.44, 'min': 0.0, 'max': 0.0}, {'current': 2497.886, 'min': 0.0, 'max': 0.0}, {'current': 3376.996, 'min': 0.0, 'max': 0.0}, {'current': 3262.597, 'min': 0.0, 'max': 0.0}, {'current': 3258.462, 'min': 0.0, 'max': 0.0}, {'current': 3243.917, 'min': 0.0, 'max': 0.0}, {'current': 3263.253, 'min': 0.0, 'max': 0.0}, {'current': 3596.819, 'min': 0.0, 'max': 0.0}, {'current': 3251.877, 'min': 0.0, 'max': 0.0}, {'current': 3453.493, 'min': 0.0, 'max': 0.0}, {'current': 3597.538, 'min': 0.0, 'max': 0.0}, {'current': 3252.456, 'min': 0.0, 'max': 0.0}, {'current': 3235.472, 'min': 0.0, 'max': 0.0}, {'current': 3258.914, 'min': 0.0, 'max': 0.0}, {'current': 3221.932, 'min': 0.0, 'max': 0.0}, {'current': 3223.056, 'min': 0.0, 'max': 0.0}, {'current': 2916.257, 'min': 0.0, 'max': 0.0}, {'current': 3221.381, 'min': 0.0, 'max': 0.0}, {'current': 3249.041, 'min': 0.0, 'max': 0.0}, {'current': 3158.094, 'min': 0.0, 'max': 0.0}, {'current': 3117.532, 'min': 0.0, 'max': 0.0}, {'current': 3180.319, 'min': 0.0, 'max': 0.0}, {'current': 3597.35, 'min': 0.0, 'max': 0.0}, {'current': 2902.289, 'min': 0.0, 'max': 0.0}, {'current': 3193.614, 'min': 0.0, 'max': 0.0}, {'current': 3132.908, 'min': 0.0, 'max': 0.0}, {'current': 3221.533, 'min': 0.0, 'max': 0.0}, {'current': 3082.135, 'min': 0.0, 'max': 0.0}, {'current': 3036.898, 'min': 0.0, 'max': 0.0}, {'current': 3596.595, 'min': 0.0, 'max': 0.0}, {'current': 3185.956, 'min': 0.0, 'max': 0.0}, {'current': 3178.002, 'min': 0.0, 'max': 0.0}, {'current': 3186.383, 'min': 0.0, 'max': 0.0}, {'current': 3138.284, 'min': 0.0, 'max': 0.0}, {'current': 2890.053, 'min': 0.0, 'max': 0.0}, {'current': 2878.155, 'min': 0.0, 'max': 0.0}, {'current': 2943.671, 'min': 0.0, 'max': 0.0}, {'current': 2924.089, 'min': 0.0, 'max': 0.0}, {'current': 2461.058, 'min': 0.0, 'max': 0.0}, {'current': 2961.225, 'min': 0.0, 'max': 0.0}, {'current': 3097.346, 'min': 0.0, 'max': 0.0}, {'current': 2638.607, 'min': 0.0, 'max': 0.0}, {'current': 3267.369, 'min': 0.0, 'max': 0.0}, {'current': 3301.989, 'min': 0.0, 'max': 0.0}, {'current': 3255.895, 'min': 0.0, 'max': 0.0}, {'current': 3305.783, 'min': 0.0, 'max': 0.0}, {'current': 3320.452, 'min': 0.0, 'max': 0.0}, {'current': 3278.446, 'min': 0.0, 'max': 0.0}, {'current': 3598.093, 'min': 0.0, 'max': 0.0}, {'current': 3266.814, 'min': 0.0, 'max': 0.0}, {'current': 3596.43, 'min': 0.0, 'max': 0.0}, {'current': 3270.648, 'min': 0.0, 'max': 0.0}, {'current': 3505.783, 'min': 0.0, 'max': 0.0}, {'current': 3297.048, 'min': 0.0, 'max': 0.0}, {'current': 3264.728, 'min': 0.0, 'max': 0.0}, {'current': 3224.761, 'min': 0.0, 'max': 0.0}, {'current': 3342.449, 'min': 0.0, 'max': 0.0}, {'current': 3342.148, 'min': 0.0, 'max': 0.0}, {'current': 3597.921, 'min': 0.0, 'max': 0.0}, {'current': 3251.183, 'min': 0.0, 'max': 0.0}, {'current': 3533.37, 'min': 0.0, 'max': 0.0}, {'current': 3446.582, 'min': 0.0, 'max': 0.0}, {'current': 3261.516, 'min': 0.0, 'max': 0.0}, {'current': 3254.535, 'min': 0.0, 'max': 0.0}, {'current': 3261.428, 'min': 0.0, 'max': 0.0}, {'current': 3262.921, 'min': 0.0, 'max': 0.0}, {'current': 3276.045, 'min': 0.0, 'max': 0.0}, {'current': 3266.835, 'min': 0.0, 'max': 0.0}, {'current': 2751.168, 'min': 0.0, 'max': 0.0}, {'current': 3167.713, 'min': 0.0, 'max': 0.0}, {'current': 3180.263, 'min': 0.0, 'max': 0.0}, {'current': 3203.948, 'min': 0.0, 'max': 0.0}, {'current': 3234.562, 'min': 0.0, 'max': 0.0}, {'current': 3486.665, 'min': 0.0, 'max': 0.0}, {'current': 2789.289, 'min': 0.0, 'max': 0.0}, {'current': 2859.707, 'min': 0.0, 'max': 0.0}, {'current': 3143.341, 'min': 0.0, 'max': 0.0}, {'current': 3089.871, 'min': 0.0, 'max': 0.0}, {'current': 2266.761, 'min': 0.0, 'max': 0.0}, {'current': 2745.206, 'min': 0.0, 'max': 0.0}, {'current': 2832.943, 'min': 0.0, 'max': 0.0}, {'current': 2820.125, 'min': 0.0, 'max': 0.0}, {'current': 3202.152, 'min': 0.0, 'max': 0.0}, {'current': 3127.913, 'min': 0.0, 'max': 0.0}, {'current': 3598.345, 'min': 0.0, 'max': 0.0}, {'current': 3220.73, 'min': 0.0, 'max': 0.0}, {'current': 3025.305, 'min': 0.0, 'max': 0.0}, {'current': 3391.839, 'min': 0.0, 'max': 0.0}, {'current': 3119.74, 'min': 0.0, 'max': 0.0}, {'current': 3231.315, 'min': 0.0, 'max': 0.0}, {'current': 2396.33, 'min': 0.0, 'max': 0.0}, {'current': 2423.797, 'min': 0.0, 'max': 0.0}, {'current': 2527.72, 'min': 0.0, 'max': 0.0}, {'current': 2467.572, 'min': 0.0, 'max': 0.0}, {'current': 2425.179, 'min': 0.0, 'max': 0.0}, {'current': 2028.21, 'min': 0.0, 'max': 0.0}, {'current': 2503.85, 'min': 0.0, 'max': 0.0}, {'current': 2452.12, 'min': 0.0, 'max': 0.0}, {'current': 3075.121, 'min': 0.0, 'max': 0.0}, {'current': 3222.332, 'min': 0.0, 'max': 0.0}, {'current': 3233.386, 'min': 0.0, 'max': 0.0}, {'current': 3200.352, 'min': 0.0, 'max': 0.0}, {'current': 3202.276, 'min': 0.0, 'max': 0.0}, {'current': 3598.346, 'min': 0.0, 'max': 0.0}, {'current': 3254.354, 'min': 0.0, 'max': 0.0}, {'current': 3379.901, 'min': 0.0, 'max': 0.0}, {'current': 3594.797, 'min': 0.0, 'max': 0.0}, {'current': 3225.912, 'min': 0.0, 'max': 0.0}, {'current': 3223.292, 'min': 0.0, 'max': 0.0}, {'current': 3225.046, 'min': 0.0, 'max': 0.0}, {'current': 3214.566, 'min': 0.0, 'max': 0.0}, {'current': 3280.304, 'min': 0.0, 'max': 0.0}, {'current': 3169.07, 'min': 0.0, 'max': 0.0}, {'current': 3225.192, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 119.94140625, 'used': 56.40110397338867}}, 'gpu': 'NVIDIA H100 80GB HBM3', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}], 'memory': {'total': 1999.9661331176758}} 2024-11-13 17:10:08,005 INFO HandlerThread:1939 [system_monitor.py:probe():224] Finished collecting system info 2024-11-13 17:10:08,005 INFO HandlerThread:1939 [system_monitor.py:probe():227] Publishing system info 2024-11-13 17:10:08,006 DEBUG HandlerThread:1939 [system_info.py:_save_conda():208] Saving list of conda packages installed into the current environment 2024-11-13 17:10:08,753 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/conda-environment.yaml 2024-11-13 17:10:22,112 DEBUG HandlerThread:1939 [system_info.py:_save_conda():220] Saving conda packages done 2024-11-13 17:10:22,112 DEBUG HandlerThread:1939 [system_info.py:_save_code():45] Saving code 2024-11-13 17:10:22,117 DEBUG HandlerThread:1939 [system_info.py:_save_code():66] Saving code done 2024-11-13 17:10:22,117 DEBUG HandlerThread:1939 [system_info.py:_save_patches():83] Saving git patches 2024-11-13 17:10:22,755 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/conda-environment.yaml 2024-11-13 17:10:22,755 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/code/qlearning_reasoning/training/sft.py 2024-11-13 17:10:22,755 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/diff.patch 2024-11-13 17:10:22,755 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/code/qlearning_reasoning/training 2024-11-13 17:10:22,756 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/code/qlearning_reasoning 2024-11-13 17:10:22,756 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/code 2024-11-13 17:10:22,764 DEBUG HandlerThread:1939 [system_info.py:_save_patches():125] Saving git patches done 2024-11-13 17:10:22,766 INFO HandlerThread:1939 [system_monitor.py:probe():229] Finished publishing system info 2024-11-13 17:10:22,767 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:10:22,768 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: keepalive 2024-11-13 17:10:22,768 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:10:22,768 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: keepalive 2024-11-13 17:10:22,768 DEBUG SenderThread:1939 [sender.py:send():382] send: files 2024-11-13 17:10:22,769 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now 2024-11-13 17:10:22,769 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file code/qlearning_reasoning/training/sft.py with policy now 2024-11-13 17:10:22,769 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file diff.patch with policy now 2024-11-13 17:10:22,773 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: python_packages 2024-11-13 17:10:22,773 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: python_packages 2024-11-13 17:10:22,774 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:10:22,774 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:10:22,777 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:10:22,923 DEBUG SenderThread:1939 [sender.py:send():382] send: telemetry 2024-11-13 17:10:22,924 DEBUG SenderThread:1939 [sender.py:send():382] send: config 2024-11-13 17:10:22,924 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:10:22,925 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 17:10:22,925 DEBUG SenderThread:1939 [sender.py:send():382] send: telemetry 2024-11-13 17:10:22,925 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 17:10:22,925 WARNING SenderThread:1939 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) 2024-11-13 17:10:22,925 DEBUG SenderThread:1939 [sender.py:send():382] send: telemetry 2024-11-13 17:10:22,925 DEBUG SenderThread:1939 [sender.py:send():382] send: telemetry 2024-11-13 17:10:22,925 DEBUG SenderThread:1939 [sender.py:send():382] send: config 2024-11-13 17:10:23,079 INFO wandb-upload_1:1939 [upload_job.py:push():131] Uploaded file /tmp/tmp6y9zx1o1wandb/yx5qp0zp-code/qlearning_reasoning/training/sft.py 2024-11-13 17:10:23,081 INFO wandb-upload_0:1939 [upload_job.py:push():131] Uploaded file /tmp/tmp6y9zx1o1wandb/ohhel8zq-wandb-metadata.json 2024-11-13 17:10:23,084 INFO wandb-upload_2:1939 [upload_job.py:push():131] Uploaded file /tmp/tmp6y9zx1o1wandb/o4jbd2x2-diff.patch 2024-11-13 17:10:23,755 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/diff.patch 2024-11-13 17:10:23,756 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/requirements.txt 2024-11-13 17:10:23,756 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:10:23,756 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/wandb-metadata.json 2024-11-13 17:10:25,756 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:10:27,927 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:10:32,928 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:10:37,774 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:10:37,774 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:10:37,775 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:10:38,878 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:10:39,761 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/config.yaml 2024-11-13 17:10:44,055 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:10:46,939 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 17:10:46,941 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 17:10:46,942 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 17:10:46,942 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 17:10:46,942 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 17:10:46,942 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 17:10:46,942 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 17:10:46,944 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 17:10:47,764 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:10:47,765 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/wandb-summary.json 2024-11-13 17:10:49,765 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:10:49,946 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:10:52,774 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:10:52,774 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:10:52,776 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:10:55,857 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:11:00,858 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:11:05,858 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:11:07,774 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:11:07,774 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:11:07,815 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:11:07,934 DEBUG SystemMonitor:1939 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-11-13 17:11:07,937 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:11:11,483 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:11:11,774 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:11:11,774 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/config.yaml 2024-11-13 17:11:16,645 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:11:21,646 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:11:22,774 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:11:22,774 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:11:22,777 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:11:26,930 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:11:31,930 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:11:35,784 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:11:37,312 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:11:37,774 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:11:37,774 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:11:37,777 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:11:37,941 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:11:42,943 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:11:47,944 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:11:52,774 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:11:52,775 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:11:52,815 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:11:53,939 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:11:57,792 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:11:59,512 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:12:04,513 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:12:07,774 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:12:07,775 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:12:07,775 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:12:07,944 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:12:09,946 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:12:14,946 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:12:18,339 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 17:12:18,341 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 17:12:18,341 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 17:12:18,341 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 17:12:18,801 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/wandb-summary.json 2024-11-13 17:12:19,801 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:12:19,952 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 17:12:19,954 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 17:12:19,955 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 17:12:19,955 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:12:19,955 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 17:12:19,956 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 17:12:19,957 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 17:12:19,957 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 17:12:19,957 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 17:12:20,802 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/wandb-summary.json 2024-11-13 17:12:21,802 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:12:22,774 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:12:22,775 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:12:22,775 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:12:25,846 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:12:30,847 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:12:35,847 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:12:37,774 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:12:37,775 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:12:37,815 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:12:37,946 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:12:40,948 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:12:43,811 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:12:46,900 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:12:47,812 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/config.yaml 2024-11-13 17:12:52,087 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:12:52,775 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:12:52,775 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:12:52,775 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:12:57,926 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:13:02,927 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:13:07,775 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:13:07,775 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:13:07,775 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:13:07,820 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:13:07,949 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:13:07,950 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:13:12,951 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:13:17,952 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:13:22,775 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:13:22,775 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:13:22,815 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:13:23,932 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:13:29,529 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:13:29,829 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:13:34,530 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:13:37,775 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:13:37,775 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:13:37,776 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:13:37,952 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:13:39,954 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:13:44,955 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:13:49,955 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:13:51,838 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:13:52,775 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:13:52,775 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:13:52,777 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:13:55,936 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:14:00,937 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:14:05,938 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:14:07,775 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:14:07,776 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:14:07,819 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:14:07,954 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:14:10,956 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:14:12,899 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 17:14:12,900 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 17:14:12,901 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 17:14:12,902 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 17:14:13,847 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/wandb-summary.json 2024-11-13 17:14:13,847 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:14:14,513 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 17:14:14,514 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 17:14:14,514 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 17:14:14,516 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 17:14:14,847 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/wandb-summary.json 2024-11-13 17:14:15,848 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:14:16,517 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:14:21,517 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:14:22,776 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:14:22,777 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:14:22,777 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:14:26,902 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:14:31,903 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:14:37,269 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:14:37,775 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:14:37,776 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:14:37,778 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:14:37,857 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:14:37,957 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:14:42,959 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:14:47,960 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:14:52,775 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:14:52,776 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:14:52,819 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:14:53,918 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:14:58,973 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:14:59,866 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:15:03,974 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:15:07,776 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:15:07,776 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:15:07,778 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:15:07,960 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:15:09,962 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:15:14,963 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:15:19,963 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:15:24,560 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:15:24,560 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:15:24,760 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:15:25,654 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:15:30,654 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:15:35,655 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:15:37,965 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:15:38,559 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:15:38,559 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:15:38,599 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:15:40,718 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:15:45,718 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:15:50,719 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:15:53,891 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:15:55,441 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:15:55,632 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:15:55,633 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:15:55,738 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:16:01,306 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:16:06,307 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:16:07,966 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:16:08,649 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:16:08,805 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:16:08,805 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:16:09,899 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:16:11,308 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:16:11,900 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:16:13,900 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:16:15,902 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:16:16,682 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:16:17,903 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:16:19,904 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:16:21,862 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:16:21,905 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:16:23,906 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:16:26,484 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:16:27,824 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:16:28,965 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:16:28,965 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:16:33,084 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:16:37,969 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:16:38,971 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:16:41,636 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:16:42,736 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:16:42,737 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:16:44,879 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:16:49,879 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:16:54,880 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:16:56,904 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:16:56,908 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:16:56,908 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:16:57,922 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:17:00,014 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:17:05,015 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:17:07,971 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:17:08,907 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:17:10,425 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:17:10,425 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:17:10,425 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:17:15,586 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:17:20,587 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:17:23,906 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:17:25,425 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:17:25,425 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:17:26,585 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:17:29,937 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:17:32,376 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:17:37,377 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:17:37,975 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:17:38,907 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:17:40,426 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:17:40,426 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:17:42,574 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:17:47,575 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:17:51,945 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:17:52,901 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:17:53,908 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:17:55,425 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:17:55,426 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:17:58,561 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:18:03,562 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:18:07,977 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:18:08,908 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:18:08,979 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:18:10,424 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:18:10,425 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:18:11,623 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 17:18:11,625 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 17:18:11,625 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 17:18:11,626 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 17:18:11,953 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/wandb-summary.json 2024-11-13 17:18:12,953 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:18:13,237 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 17:18:13,239 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 17:18:13,240 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 17:18:13,241 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 17:18:13,954 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/wandb-summary.json 2024-11-13 17:18:13,954 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:18:14,242 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:18:14,954 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:18:19,242 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:18:23,907 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:18:24,243 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:18:25,427 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:18:25,427 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:18:29,574 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:18:34,575 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:18:37,980 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:18:38,907 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:18:38,963 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:18:39,981 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:18:40,426 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:18:40,426 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:18:45,524 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:18:50,525 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:18:53,908 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:18:55,426 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:18:55,426 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:18:55,571 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:19:00,972 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:19:01,259 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:19:06,259 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:19:07,982 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:19:08,908 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:19:10,426 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:19:10,427 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:19:11,531 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:19:16,532 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:19:21,533 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:19:23,908 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:19:24,981 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:19:25,426 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:19:25,426 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:19:27,525 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:19:32,526 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:19:37,526 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:19:37,986 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:19:38,908 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:19:40,426 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:19:40,427 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:19:43,503 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:19:46,989 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:19:49,064 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:19:53,908 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:19:54,065 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:19:55,426 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:19:55,427 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:19:59,572 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:20:04,573 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:20:07,011 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 17:20:07,012 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 17:20:07,013 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 17:20:07,013 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 17:20:07,988 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:20:07,998 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/wandb-summary.json 2024-11-13 17:20:07,999 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:20:08,654 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 17:20:08,655 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 17:20:08,656 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 17:20:08,657 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 17:20:08,908 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:20:08,999 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/wandb-summary.json 2024-11-13 17:20:08,999 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:20:09,658 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:20:09,999 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:20:10,427 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:20:10,427 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:20:10,999 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:20:15,532 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:20:20,533 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:20:23,908 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:20:25,427 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:20:25,427 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:20:25,533 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:20:30,534 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:20:33,008 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:20:35,587 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:20:37,991 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:20:38,908 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:20:40,427 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:20:40,427 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:20:41,574 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:20:46,575 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:20:51,575 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:20:53,908 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:20:55,021 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:20:55,427 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:20:55,427 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:20:56,595 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:21:01,596 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:21:06,597 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:21:07,994 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:21:08,909 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:21:10,428 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:21:10,428 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:21:12,570 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:21:17,036 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:21:17,796 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:21:22,796 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:21:23,909 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:21:25,427 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:21:25,427 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:21:28,520 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:21:33,520 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:21:37,996 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:21:38,553 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:21:38,909 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:21:40,427 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:21:40,427 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:21:41,048 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:21:44,512 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:21:49,513 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:21:54,514 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:21:58,921 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:21:58,921 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:21:59,233 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:22:00,115 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:22:05,116 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:22:07,999 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:22:11,001 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:22:13,345 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:22:13,345 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:22:13,345 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:22:16,483 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:22:21,484 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:22:26,485 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:22:29,071 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:22:29,261 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:22:29,261 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:22:29,593 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:22:32,181 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:22:37,181 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:22:38,002 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:22:43,004 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:22:43,839 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:22:43,839 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:22:43,840 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:22:45,079 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:22:47,080 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:22:48,290 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:22:49,081 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:22:51,082 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:22:53,083 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:22:53,501 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:22:55,084 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:22:57,085 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:22:59,086 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:22:59,339 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:22:59,339 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:22:59,486 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:22:59,961 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:23:04,808 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:23:08,005 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:23:10,007 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:23:15,008 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:23:16,870 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:23:17,878 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:23:17,878 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:23:20,977 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:23:25,978 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:23:30,117 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:23:31,167 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:23:31,167 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:23:31,334 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:23:33,101 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:23:36,842 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:23:38,008 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:23:42,009 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:23:43,918 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:23:44,229 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:23:44,229 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:23:47,330 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:23:52,331 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:23:57,331 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:23:58,927 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:23:59,229 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:23:59,229 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:24:02,381 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:24:07,382 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:24:08,010 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:24:13,012 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:24:18,013 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:24:20,080 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:24:20,081 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:24:20,425 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:24:23,197 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:24:28,198 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:24:33,199 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:24:34,975 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:24:34,975 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:24:34,976 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:24:38,013 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:24:39,015 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:24:43,133 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:24:44,952 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:24:49,952 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:24:50,909 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:24:50,909 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:24:51,584 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:24:55,782 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:25:00,783 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:25:05,028 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:25:05,056 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:25:05,057 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:25:05,144 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:25:06,319 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:25:07,145 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:25:08,016 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:25:09,146 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:25:11,173 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:25:11,566 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:25:13,229 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:25:16,231 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:25:16,767 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:25:18,232 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:25:20,233 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:25:22,557 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:25:24,219 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:25:24,220 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:25:24,289 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:25:28,289 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:25:33,289 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:25:38,019 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:25:38,279 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:25:38,279 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:25:38,373 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:25:39,360 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:25:44,045 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:25:49,045 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:25:54,046 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:25:55,068 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:25:55,069 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:25:55,314 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:25:59,148 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:26:01,619 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 17:26:01,620 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 17:26:01,622 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 17:26:01,622 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 17:26:01,622 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 17:26:01,622 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 17:26:01,623 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 17:26:01,623 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 17:26:01,623 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 17:26:01,623 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 17:26:01,624 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 17:26:01,624 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 17:26:01,624 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 17:26:01,624 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 17:26:02,253 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/wandb-summary.json 2024-11-13 17:26:02,254 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:26:04,255 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-5_bs-64_acc-8_len-2048/wandb/run-20241113_171007-kushalarora-rvv-main-2024-11-13-16-43-09-915-0exw0n-algo-1/files/output.log 2024-11-13 17:26:04,625 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:26:08,014 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:26:08,014 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:26:08,130 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:26:08,299 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:26:10,300 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:26:15,301 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:26:20,302 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:26:23,014 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:26:23,014 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:26:23,299 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:26:26,301 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:26:31,302 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:26:36,302 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:26:38,014 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:26:38,014 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:26:38,164 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:26:38,299 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:26:42,301 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:26:47,302 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:26:52,302 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:26:53,014 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:26:53,014 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:26:53,300 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:26:58,302 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:27:03,303 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:27:08,014 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:27:08,015 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:27:08,168 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 17:27:08,299 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 17:27:09,300 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:27:14,301 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:27:19,302 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 17:27:23,014 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 17:27:23,015 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 17:27:23,300 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages