How to run with mmproj
#1
by
ShulgaSA
- opened
Can someone provide some sample code on how to run this model locally via llama_cpp_python?
I don't understand how to pick up the mmproj file along with the gguf ..
I havent used llama_cpp_python but it looks like you use "clip_model_path"
example taken from: https://github.com/abetlen/llama-cpp-python/blob/b1d23df0bbd327b774083b5cf88e67ca0dd52b92/docs/server.md
{
"model": "models/ggml_llava-v1.5-7b/ggml-model-q4_k.gguf",
"model_alias": "gpt-4-vision-preview",
"chat_format": "llava-1-5",
"clip_model_path": "models/ggml_llava-v1.5-7b/mmproj-model-f16.gguf",
"n_gpu_layers": -1,
"offload_kqv": true,
"n_threads": 12,
"n_batch": 512,
"n_ctx": 2048
}