quantization config from the arguments has no `quant_method` attribute
#1
by
endremoen
- opened
Im trying to run the provided 4 bit quantized model (Ourse/AM-Thinking-v1-mlx-4Bit) on ubuntu, but get an error:
"model = AutoModelForCausalLM.from_pretrained("Ourse/AM-Thinking-v1-mlx-4Bit", device_map='auto')
...ValueError: The model's quantization config from the arguments has no quant_method attribute. Make sure that the model has been correctly quantized". The code I run is the provided for transformers:
Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Ourse/AM-Thinking-v1-mlx-4Bit")
model = AutoModelForCausalLM.from_pretrained("Ourse/AM-Thinking-v1-mlx-4Bit")