mistralai

/

Mistral-Small-3.1-24B-Instruct-2503

like 705

Follow

Mistral AI_ 6.65k

You need 4.50.0.dev0

thanks, solved with

pip install git+https://github.com/huggingface/transformers.git

now i get:

ERROR 03-18 16:24:27 [core.py:340] ValueError: Mistral3ForConditionalGeneration has no vLLM implementation and the Transformers implementation is not compatible with vLLM. Try setting VLLM_USE_V1=0.
ERROR 03-18 16:24:27 [core.py:340] 
CRITICAL 03-18 16:24:27 [core_client.py:269] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
Killed

and I really don't understand

Are you using nightly vllm? https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503#vllm-recommended

yeah I followed the instructions, I'm just trying the script here on a A100:
https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503#offline

edit:
even setting VLLM_USE_V1=0 doesn't solve the issue but gives me the same error plus:

[rank0]:[W318 16:29:26.791873028 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

Using latest vLLM and installing transformers from source, getting the following error:

[dckr]: ERROR 03-18 09:12:51 [core.py:340]     raise ValueError(
[dckr]: ERROR 03-18 09:12:51 [core.py:340] ValueError: Mistral3ForConditionalGeneration has no vLLM implementation and the Transformers implementation is not compatible with vLLM. Try setting VLLM_USE_V1=0.

So, autoconfig isn't working yet. You need to pass the all the command line args they suggest to work.

@r3lativo , try this, had same issue about mistral3.

llm = LLM(
model="mistralai/Mistral-Small-3.1-24B-Instruct-2503",
tokenizer_mode="mistral",
config_format="mistral",
load_format="mistral",
dtype="float16"
)

edit:
dtype is optional, used it to fit VRAM