AMD | AttributeError: '_OpNamespace' '_C' object has no attribute 'awq_marlin_repack'
#1
by
djdeniro
- opened
Hey just downloading your model, but before it started, should i add more parameters for launch it with 6x24gpu?
vllm-1 | (VllmWorker rank=1 pid=413) ERROR 07-24 14:18:17 [multiproc_executor.py:511] output[e] = torch.ops._C.awq_marlin_repack(b_q_weight[e], size_k,
vllm-1 | (VllmWorker rank=1 pid=413) ERROR 07-24 14:18:17 [multiproc_executor.py:511] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
vllm-1 | (VllmWorker rank=1 pid=413) ERROR 07-24 14:18:17 [multiproc_executor.py:511] File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1267, in __getattr__
vllm-1 | (VllmWorker rank=1 pid=413) ERROR 07-24 14:18:17 [multiproc_executor.py:511] raise AttributeError(
vllm-1 | (VllmWorker rank=1 pid=413) ERROR 07-24 14:18:17 [multiproc_executor.py:511] AttributeError: '_OpNamespace' '_C' object has no attribute 'awq_marlin_repack'
Loading safetensors checkpoint shards: 96% 24/25 [00:23<00:00, 1.01it/s]
vllm-1 | [rank0]:[W724 14:18:18.278088559 ProcessGroupNCCL.cpp:1476] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
djdeniro
changed discussion title from
is it work with --pp 6?
to AMD | AttributeError: '_OpNamespace' '_C' object has no attribute 'awq_marlin_repack'
there are some tensors can't be evenly divided by 6
Please consider running it on 8 devices, each with 24GB of VRAM.
there are some tensors can't be evenly divided by 6 - this is not problem ,i t should work with -pp 3 -tp 2 or -pp 6 -tp 1. but i am agree with you, maybe it need 8xgpu without repack