New 1.66-bit TQ1_0 quant that is 162GB
#14
pinned
by
shimmyshimmer
- opened
We added a new TQ1_0 quant that is 1.66-bit and 162GB in size. For those who want it to fit exactly on some setups and some more lenient combinations
Made for setups with 192GB RAM and Ollama
shimmyshimmer
changed discussion title from
New TQ1_0 quant that is 162GB
to New 1.66-bit TQ1_0 quant that is 162GB
shimmyshimmer
pinned discussion
What’s the difference compared to IQ1_S? What layers are compressed more?
could you please share how to run it with llama cpp? Thanks.
Edit: it seems working
./llama.cpp/llama-cli \
-hf unsloth/DeepSeek-R1-0528-GGUF:TQ1_0 \
--cache-type-k q4_0 \
--threads -1 \
--n-gpu-layers 99 \
--prio 3 \
--temp 0.6 \
--top_p 0.95 \
--min_p 0.01 \
--ctx-size 16384 \
--seed 3407 \
-ot ".ffn_.*_exps.=CPU"
What’s the difference compared to IQ1_S? What layers are compressed more?
Correct but the correct layers
Is there a comparison of performance (benchmark score) between each quantized version and the original checkpoint? Thanks!