Create README.md

README.md

@@ -0,0 +1,59 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen2.5-VL-32B-Instruct
+base_model_relation: quantized
+tags:
+- Qwen
+- Qwen2.5
+- GGUF
+- quantized
+- 6-bit
+---
+
+## Llama.cpp hybrid layer quantization of Qwen2.5-VL-32B-Instruct by Alibaba
+
+Original model: https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct
+
+The hybrid quant employs different quantization levels on a per layer basis to enable
+both high performance and small file size at the same time. The quants
+employed are all K to avoid slow CPU or older GPU processing of IQ quants.  For this
+file the layer quants are as follows:
+```
+   LAYER_TYPES='[
+   [0 ,"Q4_K_M"],[1 ,"Q4_K_S"],[2 ,"Q3_K_M"],[3 ,"Q3_K_M"],[4 ,"Q3_K_M"],[5 ,"Q3_K_M"],[6 ,"Q3_K_M"],[7 ,"Q3_K_M"],
+   [8 ,"Q3_K_M"],[9 ,"Q3_K_M"],[10,"Q3_K_M"],[11,"Q3_K_M"],[12,"Q3_K_M"],[13,"Q3_K_M"],[14,"Q3_K_M"],[15,"Q3_K_M"],
+   [16,"Q3_K_L"],[17,"Q3_K_M"],[18,"Q3_K_M"],[19,"Q3_K_M"],[20,"Q3_K_L"],[21,"Q3_K_M"],[22,"Q3_K_M"],[23,"Q3_K_M"],
+   [24,"Q3_K_L"],[25,"Q3_K_L"],[26,"Q3_K_L"],[27,"Q3_K_L"],[28,"Q3_K_L"],[29,"Q3_K_L"],[30,"Q3_K_L"],[31,"Q3_K_L"],
+   [32,"Q3_K_L"],[33,"Q3_K_L"],[34,"Q3_K_L"],[35,"Q3_K_L"],[36,"Q3_K_L"],[37,"Q3_K_L"],[38,"Q3_K_L"],[39,"Q3_K_L"],
+   [40,"Q4_K_S"],[41,"Q3_K_L"],[42,"Q4_K_S"],[43,"Q3_K_L"],[44,"Q4_K_S"],[45,"Q3_K_L"],[46,"Q4_K_S"],[47,"Q3_K_L"],
+   [48,"Q4_K_S"],[49,"Q4_K_S"],[50,"Q4_K_S"],[51,"Q4_K_S"],[52,"Q4_K_M"],[53,"Q4_K_M"],[54,"Q4_K_M"],[55,"Q4_K_M"],
+   [56,"Q4_K_M"],[57,"Q4_K_M"],[58,"Q4_K_M"],[59,"Q4_K_M"],[60,"Q4_K_M"],[61,"Q5_K_S"],[62,"Q5_K_M"],[63,"Q6_K"  ]
+   ]'
+   FLAGS="--token-embedding-type Q4_K --output-tensor-type Q6_K"
+```
+Comparison:
+
+Quant |  size  |  PPL |   Comment
+---------|---------|------|-----------
+IQ4_XS  | 17.9e9 | 6.4 |  IQ4_XS with default embedding and output
+Q4_K_H |  18e9  |  6.15  | Hybrid quant with Q4_K embedding Q6_K output
+
+Usage:
+
+Qwen2.5-VL-32B-Instruct is a vision capable model.  It can be used together with its multimedia projector layers to process images
+and text inputs and generate text outputs.  The mmproj file is made available in this repository. To test vision mode follow
+the docs in the mtmd readme in the tools directory of the source tree https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd/README.md . 
+
+Benchmarks:
+
+A full set of vision benchmarks for the model will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm
+
+## Download the file from below:
+| Link | Type | Size/e9 B | Notes |
+|------|------|-----------|-------|
+| [Qwen2.5-VL-32B-Instruct.Q4_K_H.gguf](https://huggingface.co/steampunque/Qwen2.5-VL-32B-Instruct-Hybrid-GGUF/resolve/main/Qwen2.5-VL-32B-Instruct.Q4_K_H.gguf) | Q4_K_H | 17.9e9 B | ~IQ4_XS size better performance |
+| [Qwen2.5-VL-32B-Instruct.mmproj.gguf](https://huggingface.co/steampunque/Qwen2.5-VL-32B-Instruct-Hybrid-GGUF/resolve/main/Qwen2.5-VL-32B-Instruct.mmproj.gguf) | mmproj | 1.38e9 B | multimedia projector |
+ 
+A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:
+
+https://github.com/ggml-org/llama.cpp/discussions/13040