| 2025-05-05 19:10:35,744 INFO MainThread:20093 [wandb_setup.py:_flush():68] Current SDK version is 0.19.10 | |
| 2025-05-05 19:10:35,744 INFO MainThread:20093 [wandb_setup.py:_flush():68] Configure stats pid to 20093 | |
| 2025-05-05 19:10:35,744 INFO MainThread:20093 [wandb_setup.py:_flush():68] Loading settings from /root/.config/wandb/settings | |
| 2025-05-05 19:10:35,744 INFO MainThread:20093 [wandb_setup.py:_flush():68] Loading settings from /workspace/diffusion-pipe/wandb/settings | |
| 2025-05-05 19:10:35,744 INFO MainThread:20093 [wandb_setup.py:_flush():68] Loading settings from environment variables | |
| 2025-05-05 19:10:35,744 INFO MainThread:20093 [wandb_init.py:setup_run_log_directory():724] Logging user logs to /workspace/ComfyUI/models/loras/out/20250505_19-10-35/wandb/run-20250505_191035-lg5j0rns/logs/debug.log | |
| 2025-05-05 19:10:35,745 INFO MainThread:20093 [wandb_init.py:setup_run_log_directory():725] Logging internal logs to /workspace/ComfyUI/models/loras/out/20250505_19-10-35/wandb/run-20250505_191035-lg5j0rns/logs/debug-internal.log | |
| 2025-05-05 19:10:35,745 INFO MainThread:20093 [wandb_init.py:init():852] calling init triggers | |
| 2025-05-05 19:10:35,745 INFO MainThread:20093 [wandb_init.py:init():857] wandb.init called with sweep_config: {} | |
| config: {'output_dir': '/workspace/ComfyUI/models/loras/out', 'dataset': '/workspace/configs/dataset_wan.toml', 'epochs': 1000, 'micro_batch_size_per_gpu': 1, 'pipeline_stages': 1, 'gradient_accumulation_steps': 1, 'gradient_clipping': 1.0, 'warmup_steps': 40, 'activation_checkpointing': True, 'partition_method': 'parameters', 'save_dtype': torch.bfloat16, 'caching_batch_size': 1, 'steps_per_print': 1, 'video_clip_mode': 'single_beginning', 'save_every_n_epochs': 10, 'checkpoint_every_n_minutes': 120, 'blocks_to_swap': 20, 'eval_every_n_epochs': 1, 'eval_before_first_step': True, 'eval_micro_batch_size_per_gpu': 1, 'eval_gradient_accumulation_steps': 1, 'model': {'type': 'wan', 'ckpt_path': '/workspace/Wan2.1', 'transformer_path': '/workspace/ComfyUI/models/diffusion_models/wan2.1_i2v_480p_14B_bf16.safetensors', 'llm_path': '/workspace/ComfyUI/models/text_encoders/umt5-xxl-enc-bf16.safetensors', 'dtype': torch.bfloat16, 'timestep_sample_method': 'logit_normal', 'guidance': 1.0}, 'adapter': {'type': 'lora', 'rank': 32, 'dtype': torch.bfloat16, 'alpha': 32, 'dropout': 0.0}, 'optimizer': {'type': 'adamw_optimi', 'lr': 1e-05, 'betas': [0.9, 0.99], 'weight_decay': 0.01}, 'monitoring': {'enable_wandb': True, 'wandb_api_key': 'f46df1bb828b735bd22f94fff1be190ba5e046f9', 'wandb_tracker_name': 'wan-lora', 'wandb_run_name': 'wan-lora'}, 'reentrant_activation_checkpointing': False, 'logging_steps': 1, 'eval_datasets': [], 'eval_every_n_steps': None, '_wandb': {}} | |
| 2025-05-05 19:10:35,745 INFO MainThread:20093 [wandb_init.py:init():893] starting backend | |
| 2025-05-05 19:10:35,745 INFO MainThread:20093 [wandb_init.py:init():897] sending inform_init request | |
| 2025-05-05 19:10:35,748 INFO MainThread:20093 [backend.py:_multiprocessing_setup():101] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |
| 2025-05-05 19:10:35,749 INFO MainThread:20093 [wandb_init.py:init():907] backend started and connected | |
| 2025-05-05 19:10:35,751 INFO MainThread:20093 [wandb_init.py:init():1002] updated telemetry | |
| 2025-05-05 19:10:35,759 INFO MainThread:20093 [wandb_init.py:init():1026] communicating run to backend with 90.0 second timeout | |
| 2025-05-05 19:10:36,112 INFO MainThread:20093 [wandb_init.py:init():1101] starting run threads in backend | |
| 2025-05-05 19:10:36,318 INFO MainThread:20093 [wandb_run.py:_console_start():2566] atexit reg | |
| 2025-05-05 19:10:36,319 INFO MainThread:20093 [wandb_run.py:_redirect():2414] redirect: wrap_raw | |
| 2025-05-05 19:10:36,319 INFO MainThread:20093 [wandb_run.py:_redirect():2483] Wrapping output streams. | |
| 2025-05-05 19:10:36,320 INFO MainThread:20093 [wandb_run.py:_redirect():2506] Redirects installed. | |
| 2025-05-05 19:10:36,324 INFO MainThread:20093 [wandb_init.py:init():1147] run started, returning control to user process | |