7B-GGUF/README.md

AnySecret Assistant - 7B GGUF Models

Quantized GGUF versions of the 7B model for use with llama.cpp and Ollama.

Available quantizations:

• anysecret-assistant-7B-Q4_K_M.gguf - 4-bit quantization (smallest)
• anysecret-assistant-7B-Q5_K_M.gguf - 5-bit quantization (recommended)
• anysecret-assistant-7B-Q8_0.gguf - 8-bit quantization (highest quality)

Usage with Ollama

wget https://huggingface.co/anysecret-io/anysecret-assistant/resolve/main/7B-GGUF/anysecret-assistant-7B-Q5_K_M.gguf
ollama create anysecret-7b -f Modelfile