Issue with model.safetensors.index.json
Hi Kooten,
First of all, thank you so much for posting this model.
I noticed there were some issues with model.safetensors.index.json and some of the filenames.
Below is the error message I get when running it on oobabooga/text-generation-webui.
FileNotFoundError: No such file or directory: "models/Mistral-Nemo-Instruct-2407-norefuse-OAS-6.5bpw-EXL/model-00001-of-00005.safetensors"
I think the file may have been accidentally swapped with your more complete version.
I look forward to testing this again once it's updated.
I am guessing you are using the transformers model loader, select Exllamav2 instead, and probably lower max_seq_len a lot if you run out of vram. It should work.
model.safetensors.index.json is unnecessary and can be removed, it is just copied from the FP16 version of the model.
You are completely completely right. I've never tried Exllamav2 so I overlooked that when reading the model name. Thanks for pointing that out,