Any plan for IQS_XS or IQS_XXS ?
#2
by
bobchenyx
- opened
Hi there, thanks for all the great quantization versions! They are wonderfu.
I would like to inquire if there are any future plans for IQS_XS or IQS_XXS ? (a version smaller than 128G or perhaps around 100G?)
No, you can't get lower than IQ1_S, it's barely more than 1 bit per weight at that level and would be completely incomprehensible, the sizes don't exist. We'd be better off with someone trying to prune the original model to something smaller but i don't know of anyone working on that..
Thanks for the reply and explain.
I agreed with the part you said about pruning before quantization. Will take a look at this part.
Thanks again!