Mistral3ForConditionalGeneration has no vLLM implementation and the Transformers implementation is not compatible with vLLM. Try setting VLLM_USE_V1=0.
Name: transformers
Version: 4.50.0.dev0
Summary: State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow
Home-page: https://github.com/huggingface/transformers
Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors)
Author-email: [email protected]
License: Apache 2.0 License
Location: /home/deepneuronic/miniconda3/envs/vllm/lib/python3.10/site-packages
Requires: filelock, huggingface-hub, numpy, packaging, pyyaml, regex, requests, safetensors, tokenizers, tqdm
Required-by: compressed-tensors, vllm, xgrammar
but still obtaining this error while trying to load the model:
Mistral3ForConditionalGeneration has no vLLM implementation and the Transformers implementation is not compatible with vLLM. Try setting VLLM_USE_V1=0.
@pedrojfb99
You have to either install the latest nightly or build it from source. In either case, you can serve it using the commands outlined in this comment:
https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/2#67d88eba2333b06641eceef4