Higher perplexity with Q5_K_S compared to legacy Q4_0?
#6
by
stduhpf
- opened
I did some perplexity measurements over the wikitext-2-raw-v1 dataset with iq3_xs, q4_0 and q5_k_s models (all from this repo), and surprisingly the q4_0 quant was significantly ahead.
| model type | PPL |
|---|---|
| IQ3_XS | 10.0755 +/- 0.08024 |
| Q4_0 | 9.5589 +/- 0.07527 |
| Q5_K_S | 9.8540 +/- 0.08016 |
That's not normal right? The Legacy format shouldn't perform better than the larger, more advanced format.