Q4 quants
#1
by
DrRos
- opened
Hi, guys, can you, please also upload a Q4 quants? It seems model cannot be converted to gguf using latest llama.cpp (I'm getting NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre() error while running ./convert_hf_to_gguf.py --outfile ../mellum-non-quant.gguf --verbose ../Mellum-4b-sft-python/
Use llama-quantize
llama-quantize --allow-requantize Mellum-4B-SFT-Python.Q8_0.gguf Mellum-4B-SFT-Python.Q4_0.gguf Q4_0