Can't load this in Ooogabooga
Not sure if it's something I'm doing wrong, but loader throws errors. (Little LLM of horror and fiction still loads fine)
Traceback (most recent call last):
File "D:\text-generation-webui-main\text-generation-webui-main\modules\ui_model_menu.py", line 249, in load_model_wrapper
shared.model, shared.tokenizer = load_model(selected_model, loader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\text-generation-webui-main\text-generation-webui-main\modules\models.py", line 94, in load_model
output = load_func_maploader
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\text-generation-webui-main\text-generation-webui-main\modules\models.py", line 272, in llamacpp_loader
model, tokenizer = LlamaCppModel.from_pretrained(model_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\text-generation-webui-main\text-generation-webui-main\modules\llamacpp_model.py", line 103, in from_pretrained
result.model = Llama(**params)
^^^^^^^^^^^^^^^
File "D:\text-generation-webui-main\text-generation-webui-main\installer_files\env\Lib\site-packages\llama_cpp_cuda\llama.py", line 338, in init
self._model = _LlamaModel(
^^^^^^^^^^^^
File "D:\text-generation-webui-main\text-generation-webui-main\installer_files\env\Lib\site-packages\llama_cpp_cuda_internals.py", line 57, in init
raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: models\L3-Grand-HORROR-20.7B-V1.9-STABLE-Hathors-Revenge-D_AU-Q5_k_s.gguf
Corrupt download?
Did you try a different quant and/or delete and re-download?
Sounds like a download issue ; (??)
NOTE: Make sure your version of text-generation-webui- it up to date.
I ran into a similar issue. It's the Ooba installation. If you are using a runpod image, make sure to update the installation completely.
Thanks both. Running it locally, decided to try Kobold and it loads there fine.
Additional settings / running this model:
Update: I have done some research into this issue ; here is how to address it:
In "KoboldCpp" or "oobabooga/text-generation-webui" or "Silly Tavern" ;
Set the "Smoothing_factor" to 1.5 to 2.5
: in KoboldCpp -> Settings->Samplers->Advanced-> "Smooth_F"
: in text-generation-webui -> parameters -> lower right.
: In Silly Tavern this is called: "Smoothing"
NOTE: For "text-generation-webui"
-> if using GGUFs you need to use "llama_HF" (which involves downloading some config files from the SOURCE version of this model)
Source versions (and config files) of my models are here:
https://huggingface.co/collections/DavidAU/d-au-source-files-for-gguf-exl2-awq-gptq-hqq-etc-etc-66b55cb8ba25f914cbf210be
OTHER OPTIONS:
Increase rep pen to 1.1 to 1.15 (you don't need to do this if you use "smoothing_factor".
If the interface/program you are using to run AI MODELS supports "Quadratic Sampling" ("smoothing") just make the adjustment as noted.