| # medmekk/Qwen2.5-0.5B-Instruct.GGUF | |
| GGUF quantized versions of [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) | |
| ## Available Formats: | |
| - `Q2_K`: Qwen2.5-0.5B-Instruct-Q2_K.gguf | |
| - `Q3_K_S`: Qwen2.5-0.5B-Instruct-Q3_K_S.gguf | |
| - `Q3_K_M`: Qwen2.5-0.5B-Instruct-Q3_K_M.gguf | |
| - `Q3_K_L`: Qwen2.5-0.5B-Instruct-Q3_K_L.gguf | |
| - `Q4_0`: Qwen2.5-0.5B-Instruct-Q4_0.gguf | |
| - `Q4_K_S`: Qwen2.5-0.5B-Instruct-Q4_K_S.gguf | |
| - `Q4_K_M`: Qwen2.5-0.5B-Instruct-Q4_K_M.gguf | |
| - `Q5_0`: Qwen2.5-0.5B-Instruct-Q5_0.gguf | |
| - `Q5_K_S`: Qwen2.5-0.5B-Instruct-Q5_K_S.gguf | |
| - `Q5_K_M`: Qwen2.5-0.5B-Instruct-Q5_K_M.gguf | |
| - `Q6_K`: Qwen2.5-0.5B-Instruct-Q6_K.gguf | |
| - `Q8_0`: Qwen2.5-0.5B-Instruct-Q8_0.gguf | |
| - `IQ3_M_IMAT`: Qwen2.5-0.5B-Instruct-IQ3_M_imat.gguf | |
| - `IQ3_XXS_IMAT`: Qwen2.5-0.5B-Instruct-IQ3_XXS_imat.gguf | |
| - `Q4_K_M_IMAT`: Qwen2.5-0.5B-Instruct-Q4_K_M_imat.gguf | |
| - `Q4_K_S_IMAT`: Qwen2.5-0.5B-Instruct-Q4_K_S_imat.gguf | |
| - `IQ4_NL_IMAT`: Qwen2.5-0.5B-Instruct-IQ4_NL_imat.gguf | |
| - `IQ4_XS_IMAT`: Qwen2.5-0.5B-Instruct-IQ4_XS_imat.gguf | |
| - `Q5_K_M_IMAT`: Qwen2.5-0.5B-Instruct-Q5_K_M_imat.gguf | |
| - `Q5_K_S_IMAT`: Qwen2.5-0.5B-Instruct-Q5_K_S_imat.gguf | |
| ## Usage with llama.cpp: | |
| ```bash | |
| # CLI: | |
| llama-cli --hf-repo medmekk/Qwen2.5-0.5B-Instruct.GGUF --hf-file MODEL_FILE -p "Your prompt" | |
| # Server: | |
| llama-server --hf-repo medmekk/Qwen2.5-0.5B-Instruct.GGUF --hf-file MODEL_FILE -c 2048 | |
| ``` | |