| 2024-05-16 07:00:49,436 INFO StreamThr :64 [internal.py:wandb_internal():86] W&B internal server running at pid: 64, started at: 2024-05-16 07:00:49.435768 | |
| 2024-05-16 07:00:49,440 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status | |
| 2024-05-16 07:00:49,787 INFO WriterThread:64 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/run-8ecq2cdf.wandb | |
| 2024-05-16 07:00:49,787 DEBUG SenderThread:64 [sender.py:send():379] send: header | |
| 2024-05-16 07:00:49,792 DEBUG SenderThread:64 [sender.py:send():379] send: run | |
| 2024-05-16 07:00:49,941 INFO SenderThread:64 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files | |
| 2024-05-16 07:00:49,941 INFO SenderThread:64 [sender.py:_start_run_threads():1124] run started: 8ecq2cdf with start time 1715842849.438459 | |
| 2024-05-16 07:00:49,943 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-05-16 07:00:49,944 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: check_version | |
| 2024-05-16 07:00:50,039 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-05-16 07:00:50,050 DEBUG HandlerThread:64 [system_info.py:__init__():26] System info init | |
| 2024-05-16 07:00:50,050 DEBUG HandlerThread:64 [system_info.py:__init__():41] System info init done | |
| 2024-05-16 07:00:50,051 INFO HandlerThread:64 [system_monitor.py:start():194] Starting system monitor | |
| 2024-05-16 07:00:50,051 INFO SystemMonitor:64 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-05-16 07:00:50,051 INFO HandlerThread:64 [system_monitor.py:probe():214] Collecting system info | |
| 2024-05-16 07:00:50,052 INFO SystemMonitor:64 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-05-16 07:00:50,052 INFO SystemMonitor:64 [interfaces.py:start():190] Started disk monitoring | |
| 2024-05-16 07:00:50,054 INFO SystemMonitor:64 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-05-16 07:00:50,054 INFO SystemMonitor:64 [interfaces.py:start():190] Started memory monitoring | |
| 2024-05-16 07:00:50,055 INFO SystemMonitor:64 [interfaces.py:start():190] Started network monitoring | |
| 2024-05-16 07:00:50,065 DEBUG HandlerThread:64 [system_info.py:probe():150] Probing system | |
| 2024-05-16 07:00:50,067 DEBUG HandlerThread:64 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-05-16 07:00:50,067 DEBUG HandlerThread:64 [system_info.py:probe():198] Probing system done | |
| 2024-05-16 07:00:50,067 DEBUG HandlerThread:64 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-05-16T07:00:50.065608', 'startedAt': '2024-05-16T07:00:49.429145', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'cc8faff11463', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.188, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.188, 'min': 0.0, 'max': 0.0}, {'current': 2000.188, 'min': 0.0, 'max': 0.0}, {'current': 2000.188, 'min': 0.0, 'max': 0.0}, {'current': 2000.188, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5611.894302368164}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}} | |
| 2024-05-16 07:00:50,068 INFO HandlerThread:64 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-05-16 07:00:50,068 INFO HandlerThread:64 [system_monitor.py:probe():227] Publishing system info | |
| 2024-05-16 07:00:50,068 DEBUG HandlerThread:64 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment | |
| 2024-05-16 07:00:50,943 INFO Thread-12 :64 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/conda-environment.yaml | |
| 2024-05-16 07:01:05,086 ERROR HandlerThread:64 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds | |
| Traceback (most recent call last): | |
| File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda | |
| subprocess.call( | |
| File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call | |
| return p.wait(timeout=timeout) | |
| File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait | |
| return self._wait(timeout=timeout) | |
| File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait | |
| raise TimeoutExpired(self.args, timeout) | |
| subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds | |
| 2024-05-16 07:01:05,092 DEBUG HandlerThread:64 [system_info.py:_save_conda():222] Saving conda packages done | |
| 2024-05-16 07:01:05,092 INFO HandlerThread:64 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-05-16 07:01:05,106 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:01:05,106 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-05-16 07:01:05,106 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:01:05,106 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-05-16 07:01:05,106 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:01:05,106 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-05-16 07:01:05,107 DEBUG SenderThread:64 [sender.py:send():379] send: files | |
| 2024-05-16 07:01:05,107 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now | |
| 2024-05-16 07:01:05,282 INFO wandb-upload_0:64 [upload_job.py:push():131] Uploaded file /tmp/tmp15g4p1a2wandb/78g4dfty-wandb-metadata.json | |
| 2024-05-16 07:01:05,946 INFO Thread-12 :64 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-metadata.json | |
| 2024-05-16 07:01:06,113 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-05-16 07:01:06,114 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: python_packages | |
| 2024-05-16 07:01:06,116 DEBUG SenderThread:64 [sender.py:send():379] send: telemetry | |
| 2024-05-16 07:01:06,127 DEBUG SenderThread:64 [sender.py:send():379] send: config | |
| 2024-05-16 07:01:06,128 DEBUG SenderThread:64 [sender.py:send():379] send: metric | |
| 2024-05-16 07:01:06,129 DEBUG SenderThread:64 [sender.py:send():379] send: telemetry | |
| 2024-05-16 07:01:06,129 DEBUG SenderThread:64 [sender.py:send():379] send: metric | |
| 2024-05-16 07:01:06,129 WARNING SenderThread:64 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen) | |
| 2024-05-16 07:01:06,130 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:01:06,130 DEBUG SenderThread:64 [sender.py:send():379] send: telemetry | |
| 2024-05-16 07:01:06,130 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:01:06,131 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:01:06,946 INFO Thread-12 :64 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/requirements.txt | |
| 2024-05-16 07:01:07,946 INFO Thread-12 :64 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/output.log | |
| 2024-05-16 07:01:09,947 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/output.log | |
| 2024-05-16 07:01:10,543 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:01:15,544 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:01:20,550 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:01:20,952 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/config.yaml | |
| 2024-05-16 07:01:21,114 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:01:21,115 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:01:21,115 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:01:26,203 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:01:27,792 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:01:27,794 DEBUG SenderThread:64 [sender.py:send():379] send: metric | |
| 2024-05-16 07:01:27,794 DEBUG SenderThread:64 [sender.py:send():379] send: metric | |
| 2024-05-16 07:01:27,794 DEBUG SenderThread:64 [sender.py:send():379] send: metric | |
| 2024-05-16 07:01:27,794 DEBUG SenderThread:64 [sender.py:send():379] send: metric | |
| 2024-05-16 07:01:27,795 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:01:27,795 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:01:27,796 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:01:27,955 INFO Thread-12 :64 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:01:31,798 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:01:36,114 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:01:36,115 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:01:36,118 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:01:37,197 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:01:42,198 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:01:46,466 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:01:46,467 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:01:46,467 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:01:46,469 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:01:46,963 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:01:47,469 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:01:50,055 DEBUG SystemMonitor:64 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-05-16 07:01:50,057 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:01:51,114 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:01:51,115 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:01:51,118 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:01:53,247 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:01:53,966 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/config.yaml | |
| 2024-05-16 07:01:58,435 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:02:03,436 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:02:05,283 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:02:05,284 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:02:05,284 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:02:05,284 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:02:05,972 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:02:06,114 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:02:06,115 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:02:06,117 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:02:09,151 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:02:14,152 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:02:19,153 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:02:20,059 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:02:21,114 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:02:21,115 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:02:21,155 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:02:24,162 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:02:24,433 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:02:24,434 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:02:24,434 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:02:24,435 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:02:24,979 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:02:29,436 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:02:34,437 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:02:36,115 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:02:36,115 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:02:36,155 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:02:40,220 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:02:43,811 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:02:43,812 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:02:43,813 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:02:43,813 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:02:43,987 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:02:45,814 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:02:50,060 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:02:51,060 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:02:51,117 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:02:51,117 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:02:51,120 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:02:56,247 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:03:01,248 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:03:03,350 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:03:03,352 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:03:03,352 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:03:03,354 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:03:03,995 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:03:06,115 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:03:06,116 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:03:06,118 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:03:07,195 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:03:12,195 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:03:17,197 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:03:20,060 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:03:21,115 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:03:21,116 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:03:21,155 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:03:22,211 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:03:23,181 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:03:23,182 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:03:23,183 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:03:23,183 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:03:24,002 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:03:28,184 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:03:33,185 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:03:36,116 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:03:36,116 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:03:36,117 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:03:38,203 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:03:43,204 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:03:43,332 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:03:43,333 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:03:43,333 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:03:43,335 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:03:44,009 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:03:48,336 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:03:50,061 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:03:51,117 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:03:51,117 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:03:51,157 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:03:54,211 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:03:59,212 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:04:03,276 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:04:03,277 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:04:03,277 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:04:03,277 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:04:04,017 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:04:04,278 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:04:06,117 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:04:06,118 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:04:06,120 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:04:10,213 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:04:15,214 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:04:20,062 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:04:21,063 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:04:21,116 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:04:21,116 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:04:21,156 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:04:23,507 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:04:23,508 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:04:23,508 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:04:23,510 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:04:24,025 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:04:26,511 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:04:31,512 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:04:36,116 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:04:36,117 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:04:36,119 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:04:37,202 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:04:42,203 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:04:43,587 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:04:43,589 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:04:43,589 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:04:43,591 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:04:44,033 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:04:47,592 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:04:50,063 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:04:51,116 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:04:51,117 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:04:51,120 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:04:53,157 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:04:58,158 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:05:03,159 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:05:04,017 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:05:04,019 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:05:04,019 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:05:04,019 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:05:04,042 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:05:06,116 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:05:06,117 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:05:06,120 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:05:08,264 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:05:13,265 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:05:18,266 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:05:20,065 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:05:21,116 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:05:21,117 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:05:21,157 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:05:24,235 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:05:24,541 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:05:24,542 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:05:24,542 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:05:24,544 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:05:25,051 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:05:29,546 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:05:34,546 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:05:36,117 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:05:36,117 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:05:36,119 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:05:40,213 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:05:44,744 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:05:44,745 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:05:44,745 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:05:44,748 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:05:45,059 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:05:45,748 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:05:50,066 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:05:51,066 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:05:51,117 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:05:51,117 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:05:51,121 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:05:56,148 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:06:01,149 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:06:04,927 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:06:04,928 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:06:04,928 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:06:04,929 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:06:05,066 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:06:06,117 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:06:06,118 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:06:06,118 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:06:06,201 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:06:11,202 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:06:16,203 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:06:20,067 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:06:21,117 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:06:21,118 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:06:21,157 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:06:21,221 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:06:25,264 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:06:25,266 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:06:25,266 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:06:25,267 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:06:26,075 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:06:26,268 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:06:31,269 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:06:36,117 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:06:36,118 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:06:36,118 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:06:37,216 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:06:42,217 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:06:45,384 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:06:45,385 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:06:45,385 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:06:45,386 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:06:46,083 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:06:47,386 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:06:50,068 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:06:51,117 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:06:51,118 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:06:51,118 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:06:53,222 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:06:58,223 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:07:03,224 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:07:05,752 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:07:05,753 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:07:05,753 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:07:05,754 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:07:06,090 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:07:06,119 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:07:06,120 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:07:06,122 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:07:09,202 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:07:14,203 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:07:19,204 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:07:20,069 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:07:21,118 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:07:21,118 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:07:21,158 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:07:24,227 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:07:26,067 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:07:26,068 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:07:26,068 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:07:26,069 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:07:26,099 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:07:30,070 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:07:35,071 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:07:36,118 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:07:36,119 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:07:36,119 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:07:40,238 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:07:45,240 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:07:46,449 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:07:46,450 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:07:46,451 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:07:46,451 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:07:47,107 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:07:50,070 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:07:51,071 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:07:51,118 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:07:51,119 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:07:51,121 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:07:56,168 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:08:01,168 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:08:06,118 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:08:06,119 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:08:06,159 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:08:06,237 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:08:07,128 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:08:07,129 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:08:07,129 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:08:07,129 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:08:08,115 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:08:12,130 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:08:17,131 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:08:20,070 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:08:21,118 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:08:21,119 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:08:21,122 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:08:22,255 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:08:27,256 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:08:27,319 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:08:27,320 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:08:27,320 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:08:27,321 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:08:28,125 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:08:32,322 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:08:36,118 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:08:36,119 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:08:36,119 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:08:38,194 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:08:43,195 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:08:47,465 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:08:47,466 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:08:47,466 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:08:47,466 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:08:48,133 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:08:48,467 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:08:50,071 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:08:51,119 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:08:51,119 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:08:51,121 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:08:54,171 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:08:59,172 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:09:04,173 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:09:06,119 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:09:06,119 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:09:06,159 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:09:07,813 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:09:07,814 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:09:07,814 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:09:07,814 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:09:08,140 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:09:09,815 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:09:14,816 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:09:19,817 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:09:20,072 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:09:21,119 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:09:21,120 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:09:21,122 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:09:25,223 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:09:28,544 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:09:28,545 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:09:28,545 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:09:28,546 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:09:29,149 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:09:30,546 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:09:35,547 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:09:36,119 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:09:36,120 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:09:36,123 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:09:41,221 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:09:46,222 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:09:48,805 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:09:48,806 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:09:48,806 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:09:48,806 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:09:49,157 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:09:50,073 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:09:51,119 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:09:51,120 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:09:51,120 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:09:51,239 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:09:56,240 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:10:01,240 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:10:06,119 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:10:06,120 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:10:06,160 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:10:07,195 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:10:09,026 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:10:09,027 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:10:09,027 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:10:09,028 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:10:09,165 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:10:13,029 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:10:18,030 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:10:20,079 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:10:21,120 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:10:21,120 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:10:21,123 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:10:23,227 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:10:28,228 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:10:29,221 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:10:29,222 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:10:29,223 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:10:29,223 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:10:30,172 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:10:34,224 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:10:36,120 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:10:36,120 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:10:36,124 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:10:40,186 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:10:45,187 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:10:49,286 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:10:49,287 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:10:49,287 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:10:49,288 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:10:50,075 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:10:50,179 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:10:51,076 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:10:51,120 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:10:51,121 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:10:51,121 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:10:56,147 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:11:01,148 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:11:06,120 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:11:06,121 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:11:06,160 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:11:06,207 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:11:09,417 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:11:09,418 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:11:09,418 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:11:09,420 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:11:10,186 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:11:11,421 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:11:16,422 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:11:20,077 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:11:21,120 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:11:21,121 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:11:21,123 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:11:22,156 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:11:27,157 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:11:30,059 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:11:30,060 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:11:30,060 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:11:30,060 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:11:30,193 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:11:33,061 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:11:36,120 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:11:36,121 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:11:36,125 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:11:38,224 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:11:43,225 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:11:48,226 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:11:50,078 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:11:50,320 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:11:50,321 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:11:50,322 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:11:50,322 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:11:51,120 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:11:51,121 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:11:51,125 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:11:51,200 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:11:54,155 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:11:59,156 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:12:04,157 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:12:06,121 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:12:06,121 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:12:06,161 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:12:09,238 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:12:10,490 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:12:10,491 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:12:10,491 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:12:10,492 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:12:11,207 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:12:14,493 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:12:19,494 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:12:20,079 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:12:21,121 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:12:21,122 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:12:21,122 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:12:25,210 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:12:30,212 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:12:30,795 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:12:30,797 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:12:30,797 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:12:30,797 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:12:31,215 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:12:35,798 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:12:36,121 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:12:36,122 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:12:36,124 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:12:41,231 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:12:46,231 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:12:50,080 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:12:51,087 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:12:51,089 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:12:51,089 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:12:51,089 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:12:51,128 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:12:51,128 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:12:51,129 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:12:51,222 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:12:51,233 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:12:56,234 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:13:01,235 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:13:06,121 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:13:06,122 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:13:06,162 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:13:07,173 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:13:11,270 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:13:11,271 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:13:11,272 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:13:11,272 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:13:12,231 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:13:12,273 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:13:17,274 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:13:20,081 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:13:21,121 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:13:21,122 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:13:21,122 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:13:23,189 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:13:28,190 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:13:31,420 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:13:31,421 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:13:31,421 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:13:31,422 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:13:32,238 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:13:33,423 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:13:36,122 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:13:36,122 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:13:36,124 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:13:39,228 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:13:44,229 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:13:49,230 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:13:50,082 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:13:51,122 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:13:51,122 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:13:51,162 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:13:51,624 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:13:51,625 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:13:51,625 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:13:51,626 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:13:52,246 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:13:54,627 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:13:59,628 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:14:04,629 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:14:06,122 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:14:06,122 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:14:06,125 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:14:10,171 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:14:12,027 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:14:12,028 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:14:12,028 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:14:12,029 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:14:12,253 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:14:16,029 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:14:20,083 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:14:21,084 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:14:21,122 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:14:21,123 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:14:21,123 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:14:26,192 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:14:31,193 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:14:32,465 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:14:32,466 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:14:32,466 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:14:32,467 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:14:33,261 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:14:36,122 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:14:36,123 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:14:36,123 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:14:36,221 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:14:41,222 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:14:46,223 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:14:50,084 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:14:51,122 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:14:51,123 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:14:51,163 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:14:51,247 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:14:53,290 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:14:53,291 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:14:53,292 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:14:53,292 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:14:54,270 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:14:56,293 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:15:01,294 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:15:06,122 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:15:06,123 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:15:06,123 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:15:07,200 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:15:12,201 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:15:13,941 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:15:13,942 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:15:13,942 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:15:13,943 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:15:14,277 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:15:17,944 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:15:20,085 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:15:21,123 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:15:21,123 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:15:21,126 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:15:23,197 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:15:28,198 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:15:33,199 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:15:34,476 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:15:34,477 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:15:34,477 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:15:34,477 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:15:35,285 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:15:36,123 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:15:36,123 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:15:36,126 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:15:39,196 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:15:44,197 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:15:49,198 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:15:50,085 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:15:51,123 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:15:51,124 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:15:51,163 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:15:55,183 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:15:55,230 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:15:55,231 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:15:55,232 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:15:55,232 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:15:55,292 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:16:00,233 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:16:05,234 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:16:06,123 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:16:06,124 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:16:06,124 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:16:11,181 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:16:15,644 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:16:15,645 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:16:15,645 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:16:15,646 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:16:16,063 DEBUG SenderThread:64 [sender.py:send():379] send: telemetry | |
| 2024-05-16 07:16:16,063 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:16:16,063 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-05-16 07:16:16,065 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:16:16,065 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:16:16,065 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:16:16,066 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:16:16,066 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:16:16,066 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:16:16,067 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:16:16,067 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:16:16,067 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:16:16,068 DEBUG SenderThread:64 [sender.py:send():379] send: history | |
| 2024-05-16 07:16:16,068 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: summary_record | |
| 2024-05-16 07:16:16,068 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: pause | |
| 2024-05-16 07:16:16,068 INFO HandlerThread:64 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-05-16 07:16:16,068 INFO HandlerThread:64 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-05-16 07:16:16,069 INFO HandlerThread:64 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-05-16 07:16:16,069 DEBUG SystemMonitor:64 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-05-16 07:16:16,069 INFO HandlerThread:64 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-05-16 07:16:16,069 INFO SenderThread:64 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-05-16 07:16:16,070 DEBUG SystemMonitor:64 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-05-16 07:16:16,082 INFO HandlerThread:64 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-05-16 07:16:16,083 INFO HandlerThread:64 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-05-16 07:16:16,083 INFO HandlerThread:64 [interfaces.py:finish():202] Joined network monitor | |
| 2024-05-16 07:16:16,083 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: resume | |
| 2024-05-16 07:16:16,083 INFO HandlerThread:64 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-05-16 07:16:16,083 INFO HandlerThread:64 [system_monitor.py:start():194] Starting system monitor | |
| 2024-05-16 07:16:16,084 INFO SystemMonitor:64 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-05-16 07:16:16,084 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:16:16,084 INFO SystemMonitor:64 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-05-16 07:16:16,085 INFO SystemMonitor:64 [interfaces.py:start():190] Started disk monitoring | |
| 2024-05-16 07:16:16,086 INFO SystemMonitor:64 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-05-16 07:16:16,086 INFO SystemMonitor:64 [interfaces.py:start():190] Started memory monitoring | |
| 2024-05-16 07:16:16,087 INFO SystemMonitor:64 [interfaces.py:start():190] Started network monitoring | |
| 2024-05-16 07:16:16,259 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:16:16,301 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/wandb-summary.json | |
| 2024-05-16 07:16:17,301 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/output.log | |
| 2024-05-16 07:16:18,185 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: pause | |
| 2024-05-16 07:16:18,185 INFO HandlerThread:64 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-05-16 07:16:18,185 INFO HandlerThread:64 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-05-16 07:16:18,185 DEBUG SystemMonitor:64 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-05-16 07:16:18,186 DEBUG SystemMonitor:64 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-05-16 07:16:18,186 DEBUG SystemMonitor:64 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-05-16 07:16:18,186 INFO HandlerThread:64 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-05-16 07:16:18,187 INFO HandlerThread:64 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-05-16 07:16:18,198 INFO HandlerThread:64 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-05-16 07:16:18,198 INFO HandlerThread:64 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-05-16 07:16:18,198 INFO HandlerThread:64 [interfaces.py:finish():202] Joined network monitor | |
| 2024-05-16 07:16:18,199 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:16:21,123 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:16:21,124 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:16:21,124 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:16:22,192 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:16:27,193 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:16:32,194 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:16:36,123 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:16:36,124 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:16:36,164 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:16:38,197 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:16:39,309 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/config.yaml | |
| 2024-05-16 07:16:43,373 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:16:48,374 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:16:51,123 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:16:51,124 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:16:51,164 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:16:54,183 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:16:59,184 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:17:04,184 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:17:06,124 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:17:06,124 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:17:06,164 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:17:10,172 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:17:15,173 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:17:20,174 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:17:21,124 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:17:21,124 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:17:21,154 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:17:26,156 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:17:31,157 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:17:36,124 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:17:36,125 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:17:36,164 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:17:36,165 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:17:41,166 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:17:46,168 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:17:51,124 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:17:51,125 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:17:51,165 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:17:51,211 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:17:56,212 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:18:01,214 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:18:06,125 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:18:06,125 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:18:06,165 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:18:06,263 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:18:11,264 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:18:16,266 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:18:21,125 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:18:21,125 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:18:21,165 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:18:22,258 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:18:27,259 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:18:32,260 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:18:36,125 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:18:36,125 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:18:36,165 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:18:38,244 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:18:43,245 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:18:48,246 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:18:51,125 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:18:51,125 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:18:51,165 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:18:53,264 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:18:56,637 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: resume | |
| 2024-05-16 07:18:56,637 INFO HandlerThread:64 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-05-16 07:18:56,637 INFO HandlerThread:64 [system_monitor.py:start():194] Starting system monitor | |
| 2024-05-16 07:18:56,638 INFO SystemMonitor:64 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-05-16 07:18:56,638 INFO SystemMonitor:64 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-05-16 07:18:56,639 INFO SystemMonitor:64 [interfaces.py:start():190] Started disk monitoring | |
| 2024-05-16 07:18:56,639 INFO SystemMonitor:64 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-05-16 07:18:56,640 INFO SystemMonitor:64 [interfaces.py:start():190] Started memory monitoring | |
| 2024-05-16 07:18:56,641 INFO SystemMonitor:64 [interfaces.py:start():190] Started network monitoring | |
| 2024-05-16 07:18:56,684 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: pause | |
| 2024-05-16 07:18:56,685 INFO HandlerThread:64 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-05-16 07:18:56,685 INFO HandlerThread:64 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-05-16 07:18:56,685 DEBUG SystemMonitor:64 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-05-16 07:18:56,685 DEBUG SystemMonitor:64 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-05-16 07:18:56,685 DEBUG SystemMonitor:64 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-05-16 07:18:56,686 INFO HandlerThread:64 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-05-16 07:18:56,686 INFO HandlerThread:64 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-05-16 07:18:56,696 INFO HandlerThread:64 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-05-16 07:18:56,696 INFO HandlerThread:64 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-05-16 07:18:56,697 INFO HandlerThread:64 [interfaces.py:finish():202] Joined network monitor | |
| 2024-05-16 07:18:56,697 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:18:58,698 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:19:03,699 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:19:06,125 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:19:06,126 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:19:06,126 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:19:09,255 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:19:14,257 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:19:19,258 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:19:21,125 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:19:21,126 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:19:21,166 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:19:25,192 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:19:30,193 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:19:35,194 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:19:36,125 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:19:36,126 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:19:36,166 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:19:40,228 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:19:43,461 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: resume | |
| 2024-05-16 07:19:43,461 INFO HandlerThread:64 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-05-16 07:19:43,461 INFO HandlerThread:64 [system_monitor.py:start():194] Starting system monitor | |
| 2024-05-16 07:19:43,461 INFO SystemMonitor:64 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-05-16 07:19:43,462 INFO SystemMonitor:64 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-05-16 07:19:43,462 INFO SystemMonitor:64 [interfaces.py:start():190] Started disk monitoring | |
| 2024-05-16 07:19:43,462 INFO SystemMonitor:64 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-05-16 07:19:43,463 INFO SystemMonitor:64 [interfaces.py:start():190] Started memory monitoring | |
| 2024-05-16 07:19:43,463 INFO SystemMonitor:64 [interfaces.py:start():190] Started network monitoring | |
| 2024-05-16 07:19:43,508 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: pause | |
| 2024-05-16 07:19:43,508 INFO HandlerThread:64 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-05-16 07:19:43,508 INFO HandlerThread:64 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-05-16 07:19:43,508 DEBUG SystemMonitor:64 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-05-16 07:19:43,509 DEBUG SystemMonitor:64 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-05-16 07:19:43,509 INFO HandlerThread:64 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-05-16 07:19:43,509 DEBUG SystemMonitor:64 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-05-16 07:19:43,509 INFO HandlerThread:64 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-05-16 07:19:43,521 INFO HandlerThread:64 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-05-16 07:19:43,522 INFO HandlerThread:64 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-05-16 07:19:43,522 INFO HandlerThread:64 [interfaces.py:finish():202] Joined network monitor | |
| 2024-05-16 07:19:43,522 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:19:45,523 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:19:50,524 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:19:51,126 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:19:51,126 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:19:51,128 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:19:54,271 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: resume | |
| 2024-05-16 07:19:54,271 INFO HandlerThread:64 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-05-16 07:19:54,271 INFO HandlerThread:64 [system_monitor.py:start():194] Starting system monitor | |
| 2024-05-16 07:19:54,271 INFO SystemMonitor:64 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-05-16 07:19:54,272 INFO SystemMonitor:64 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-05-16 07:19:54,272 INFO SystemMonitor:64 [interfaces.py:start():190] Started disk monitoring | |
| 2024-05-16 07:19:54,272 INFO SystemMonitor:64 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-05-16 07:19:54,273 INFO SystemMonitor:64 [interfaces.py:start():190] Started memory monitoring | |
| 2024-05-16 07:19:54,274 INFO SystemMonitor:64 [interfaces.py:start():190] Started network monitoring | |
| 2024-05-16 07:19:54,320 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: pause | |
| 2024-05-16 07:19:54,320 INFO HandlerThread:64 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-05-16 07:19:54,320 INFO HandlerThread:64 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-05-16 07:19:54,321 DEBUG SystemMonitor:64 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-05-16 07:19:54,321 DEBUG SystemMonitor:64 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-05-16 07:19:54,321 INFO HandlerThread:64 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-05-16 07:19:54,321 DEBUG SystemMonitor:64 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-05-16 07:19:54,322 INFO HandlerThread:64 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-05-16 07:19:54,333 INFO HandlerThread:64 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-05-16 07:19:54,333 INFO HandlerThread:64 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-05-16 07:19:54,333 INFO HandlerThread:64 [interfaces.py:finish():202] Joined network monitor | |
| 2024-05-16 07:19:54,334 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:19:56,335 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:20:01,336 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:20:06,126 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:20:06,127 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-05-16 07:20:06,127 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:20:06,269 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: resume | |
| 2024-05-16 07:20:06,269 INFO HandlerThread:64 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-05-16 07:20:06,269 INFO HandlerThread:64 [system_monitor.py:start():194] Starting system monitor | |
| 2024-05-16 07:20:06,269 INFO SystemMonitor:64 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-05-16 07:20:06,269 INFO SystemMonitor:64 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-05-16 07:20:06,270 INFO SystemMonitor:64 [interfaces.py:start():190] Started disk monitoring | |
| 2024-05-16 07:20:06,271 INFO SystemMonitor:64 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-05-16 07:20:06,274 INFO SystemMonitor:64 [interfaces.py:start():190] Started memory monitoring | |
| 2024-05-16 07:20:06,275 INFO SystemMonitor:64 [interfaces.py:start():190] Started network monitoring | |
| 2024-05-16 07:20:06,319 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: pause | |
| 2024-05-16 07:20:06,319 INFO HandlerThread:64 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-05-16 07:20:06,319 INFO HandlerThread:64 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-05-16 07:20:06,320 DEBUG SystemMonitor:64 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-05-16 07:20:06,320 DEBUG SystemMonitor:64 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-05-16 07:20:06,321 DEBUG SystemMonitor:64 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-05-16 07:20:06,322 INFO HandlerThread:64 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-05-16 07:20:06,323 INFO HandlerThread:64 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-05-16 07:20:06,332 INFO HandlerThread:64 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-05-16 07:20:06,333 INFO HandlerThread:64 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-05-16 07:20:06,333 INFO HandlerThread:64 [interfaces.py:finish():202] Joined network monitor | |
| 2024-05-16 07:20:06,333 DEBUG SenderThread:64 [sender.py:send():379] send: stats | |
| 2024-05-16 07:20:07,334 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:20:08,390 INFO Thread-12 :64 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240516_070049-8ecq2cdf/files/output.log | |
| 2024-05-16 07:20:12,335 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:20:17,336 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-05-16 07:20:19,143 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: resume | |
| 2024-05-16 07:20:19,143 INFO HandlerThread:64 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-05-16 07:20:19,143 INFO HandlerThread:64 [system_monitor.py:start():194] Starting system monitor | |
| 2024-05-16 07:20:19,144 INFO SystemMonitor:64 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-05-16 07:20:19,144 INFO SystemMonitor:64 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-05-16 07:20:19,144 INFO SystemMonitor:64 [interfaces.py:start():190] Started disk monitoring | |
| 2024-05-16 07:20:19,145 INFO SystemMonitor:64 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-05-16 07:20:19,146 INFO SystemMonitor:64 [interfaces.py:start():190] Started memory monitoring | |
| 2024-05-16 07:20:19,147 INFO SystemMonitor:64 [interfaces.py:start():190] Started network monitoring | |
| 2024-05-16 07:20:21,126 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-05-16 07:20:21,126 DEBUG SenderThread:64 [sender.py:send_request():406] send_request: stop_status | |
| 2024-05-16 07:20:21,129 DEBUG HandlerThread:64 [handler.py:handle_request():146] handle_request: internal_messages | |