4 gpu ERROR
(VllmWorker rank=3 pid=1825672) INFO 05-03 08:11:22 [parallel_state.py:1004] rank 3 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 3
(VllmWorker rank=2 pid=1825671) INFO 05-03 08:11:22 [parallel_state.py:1004] rank 2 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 2
(VllmWorker rank=1 pid=1825670) INFO 05-03 08:11:22 [parallel_state.py:1004] rank 1 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 1
(VllmWorker rank=0 pid=1825669) INFO 05-03 08:11:22 [parallel_state.py:1004] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 0
(VllmWorker rank=3 pid=1825672) INFO 05-03 08:11:22 [cuda.py:221] Using Flash Attention backend on V1 engine.
(VllmWorker rank=0 pid=1825669) INFO 05-03 08:11:22 [cuda.py:221] Using Flash Attention backend on V1 engine.
(VllmWorker rank=1 pid=1825670) INFO 05-03 08:11:22 [cuda.py:221] Using Flash Attention backend on V1 engine.
(VllmWorker rank=2 pid=1825671) INFO 05-03 08:11:22 [cuda.py:221] Using Flash Attention backend on V1 engine.
(VllmWorker rank=3 pid=1825672) WARNING 05-03 08:11:22 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=1 pid=1825670) WARNING 05-03 08:11:22 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=0 pid=1825669) WARNING 05-03 08:11:22 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=2 pid=1825671) WARNING 05-03 08:11:22 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=3 pid=1825672) INFO 05-03 08:11:22 [gpu_model_runner.py:1329] Starting to load model /data01/radmin/sd1/huggingface/hub/Qwen_Qwen3-30B-A3B-FP8/...
(VllmWorker rank=3 pid=1825672) WARNING 05-03 08:11:22 [utils.py:168] The model class Qwen3MoeForCausalLM has not defined packed_modules_mapping, this may lead to incorrect mapping of quantized or ignored modules
(VllmWorker rank=0 pid=1825669) INFO 05-03 08:11:22 [gpu_model_runner.py:1329] Starting to load model /data01/radmin/sd1/huggingface/hub/Qwen_Qwen3-30B-A3B-FP8/...
(VllmWorker rank=0 pid=1825669) WARNING 05-03 08:11:22 [utils.py:168] The model class Qwen3MoeForCausalLM has not defined packed_modules_mapping, this may lead to incorrect mapping of quantized or ignored modules
(VllmWorker rank=2 pid=1825671) INFO 05-03 08:11:22 [gpu_model_runner.py:1329] Starting to load model /data01/radmin/sd1/huggingface/hub/Qwen_Qwen3-30B-A3B-FP8/...
(VllmWorker rank=1 pid=1825670) INFO 05-03 08:11:22 [gpu_model_runner.py:1329] Starting to load model /data01/radmin/sd1/huggingface/hub/Qwen_Qwen3-30B-A3B-FP8/...
(VllmWorker rank=2 pid=1825671) WARNING 05-03 08:11:22 [utils.py:168] The model class Qwen3MoeForCausalLM has not defined packed_modules_mapping, this may lead to incorrect mapping of quantized or ignored modules
(VllmWorker rank=1 pid=1825670) WARNING 05-03 08:11:22 [utils.py:168] The model class Qwen3MoeForCausalLM has not defined packed_modules_mapping, this may lead to incorrect mapping of quantized or ignored modules
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] WorkerProc failed to start.
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] Traceback (most recent call last):
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 409, in worker_main
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] worker = WorkerProc(*args, **kwargs)
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 306, in init
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] self.worker.load_model()
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 162, in load_model
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] self.model_runner.load_model()
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1332, in load_model
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] self.model = get_model(vllm_config=self.vllm_config)
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/model_loader/init.py", line 14, in get_model
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] return loader.load_model(vllm_config=vllm_config)
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/model_loader/loader.py", line 452, in load_model
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] model = _initialize_model(vllm_config=vllm_config)
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/model_loader/loader.py", line 133, in _initialize_model
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] return model_class(vllm_config=vllm_config, prefix=prefix)
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_moe.py", line 488, in init
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] self.model = Qwen3MoeModel(vllm_config=vllm_config,
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 151, in init
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_moe.py", line 334, in init
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] self.start_layer, self.end_layer, self.layers = make_layers(
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] ^^^^^^^^^^^^
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 610, in make_layers
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_moe.py", line 336, in
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] lambda prefix: Qwen3MoeDecoderLayer(config=config,
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_moe.py", line 278, in init
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] self.mlp = Qwen3MoeSparseMoeBlock(config=config,
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_moe.py", line 113, in init
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] self.experts = FusedMoE(num_experts=config.num_experts,
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/layer.py", line 517, in init
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] self.quant_method.create_weights(layer=self, **moe_quant_params)
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/fp8.py", line 477, in create_weights
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] raise ValueError(
(VllmWorker rank=1 pid=1825670) ERROR 05-03 08:11:22 [multiproc_executor.py:435] ValueError: The output_size of gate's and up's weight = 192 is not divisible by weight quantization block_n = 128.
[rank0]:[W503 08:11:23.111382367 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
ERROR 05-03 08:11:24 [core.py:396] EngineCore failed to start.
ERROR 05-03 08:11:24 [core.py:396] Traceback (most recent call last):
ERROR 05-03 08:11:24 [core.py:396] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 387, in run_engine_core
ERROR 05-03 08:11:24 [core.py:396] engine_core = EngineCoreProc(*args, **kwargs)
ERROR 05-03 08:11:24 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-03 08:11:24 [core.py:396] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 329, in init
ERROR 05-03 08:11:24 [core.py:396] super().init(vllm_config, executor_class, log_stats,
ERROR 05-03 08:11:24 [core.py:396] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 64, in init
ERROR 05-03 08:11:24 [core.py:396] self.model_executor = executor_class(vllm_config)
ERROR 05-03 08:11:24 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-03 08:11:24 [core.py:396] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 52, in init
ERROR 05-03 08:11:24 [core.py:396] self._init_executor()
ERROR 05-03 08:11:24 [core.py:396] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 91, in _init_executor
ERROR 05-03 08:11:24 [core.py:396] self.workers = WorkerProc.wait_for_ready(unready_workers)
ERROR 05-03 08:11:24 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-03 08:11:24 [core.py:396] File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 370, in wait_for_ready
ERROR 05-03 08:11:24 [core.py:396] raise e from None
ERROR 05-03 08:11:24 [core.py:396] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
Process EngineCore_0:
Traceback (most recent call last):
File "/data01/anaconda3/envs/vllm/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/data01/anaconda3/envs/vllm/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 400, in run_engine_core
raise e
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 387, in run_engine_core
engine_core = EngineCoreProc(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 329, in init
super().init(vllm_config, executor_class, log_stats,
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 64, in init
self.model_executor = executor_class(vllm_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 52, in init
self._init_executor()
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 91, in _init_executor
self.workers = WorkerProc.wait_for_ready(unready_workers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 370, in wait_for_ready
raise e from None
Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
Traceback (most recent call last):
File "/data01/anaconda3/envs/vllm/lib/python3.12/weakref.py", line 666, in _exitfunc
f()
File "/data01/anaconda3/envs/vllm/lib/python3.12/weakref.py", line 590, in call
return info.func(*info.args, **(info.kwargs or {}))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 228, in shutdown
for w in self.workers:
^^^^^^^^^^^^
AttributeError: 'MultiprocExecutor' object has no attribute 'workers'
Traceback (most recent call last):
File "/data01/anaconda3/envs/vllm/bin/vllm", line 8, in
sys.exit(main())
^^^^^^
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 53, in main
args.dispatch_function(args)
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 27, in cmd
uvloop.run(run_server(args))
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/uvloop/init.py", line 109, in run
return __asyncio.run(
^^^^^^^^^^^^^^
File "/data01/anaconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 194, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/data01/anaconda3/envs/vllm/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/uvloop/init.py", line 61, in wrapper
return await main
^^^^^^^^^^
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1078, in run_server
async with build_async_engine_client(args) as engine_client:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data01/anaconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data01/anaconda3/envs/vllm/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 178, in build_async_engine_client_from_engine_args
async_llm = AsyncLLM.from_vllm_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 150, in from_vllm_config
return cls(
^^^^
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 118, in init
self.engine_core = core_client_class(
^^^^^^^^^^^^^^^^^^
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 642, in init
super().init(
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 398, in init
self._wait_for_engine_startup()
File "/data01/anaconda3/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 430, in _wait_for_engine_startup
raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above.
/data01/anaconda3/envs/vllm/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '