Quantized models with vision included?
I have not found any quantized models yet that retain vision component. Is that technically possible and are there examples?
I have not found any quantized models yet that retain vision component. Is that technically possible and are there examples?
I'm not sure about other quant formats, but about GGUF I've read that the llamacpp author gave up on the support for vision, so naturally if the format itself doesn't support it, there's no way you could use it. I believe LM Studio still supports some LLMs with vision, but the support is probably limited to older models from the time when the support was actively maintained in llamacpp itself.
That's not entirely true, see https://github.com/ggml-org/llama.cpp/issues/8010
There's still a ton of refactoring to do though
Yeah vision is far from dead but is definitely a struggle