3B-GGUF/README.md

AnySecret Assistant - 3B GGUF Models

Quantized GGUF versions of the 3B model for use with llama.cpp and Ollama.

Available quantizations:

• anysecret-assistant-3B-Q4_K_M.gguf - 4-bit quantization (smallest)
• anysecret-assistant-3B-Q5_K_M.gguf - 5-bit quantization (recommended)
• anysecret-assistant-3B-Q8_0.gguf - 8-bit quantization (highest quality)

Usage with Ollama

wget https://huggingface.co/anysecret-io/anysecret-assistant/resolve/main/3B-GGUF/anysecret-assistant-3B-Q5_K_M.gguf
ollama create anysecret-3b -f Modelfile