bartowski commited on
Commit
6f1e022
·
verified ·
1 Parent(s): 050cde1

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -14
README.md CHANGED
@@ -59,33 +59,19 @@ Run them directly with [llama.cpp](https://github.com/ggerganov/llama.cpp), or a
59
  | -------- | ---------- | --------- | ----- | ----------- |
60
  | [Mistral-Small-3.1-24B-Instruct-2503-bf16.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-bf16.gguf) | bf16 | 47.15GB | false | Full BF16 weights. |
61
  | [Mistral-Small-3.1-24B-Instruct-2503-Q8_0.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q8_0.gguf) | Q8_0 | 25.05GB | false | Extremely high quality, generally unneeded but max available quant. |
62
- | [Mistral-Small-3.1-24B-Instruct-2503-Q8_0.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q8_0.gguf) | Q8_0 | 25.05GB | false | Extremely high quality, generally unneeded but max available quant. |
63
- | [Mistral-Small-3.1-24B-Instruct-2503-Q6_K_L.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q6_K_L.gguf) | Q6_K_L | 19.67GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
64
  | [Mistral-Small-3.1-24B-Instruct-2503-Q6_K_L.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q6_K_L.gguf) | Q6_K_L | 19.67GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
65
  | [Mistral-Small-3.1-24B-Instruct-2503-Q6_K.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q6_K.gguf) | Q6_K | 19.35GB | false | Very high quality, near perfect, *recommended*. |
66
- | [Mistral-Small-3.1-24B-Instruct-2503-Q6_K.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q6_K.gguf) | Q6_K | 19.35GB | false | Very high quality, near perfect, *recommended*. |
67
- | [Mistral-Small-3.1-24B-Instruct-2503-Q5_K_L.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q5_K_L.gguf) | Q5_K_L | 17.18GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
68
  | [Mistral-Small-3.1-24B-Instruct-2503-Q5_K_L.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q5_K_L.gguf) | Q5_K_L | 17.18GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
69
  | [Mistral-Small-3.1-24B-Instruct-2503-Q5_K_M.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q5_K_M.gguf) | Q5_K_M | 16.76GB | false | High quality, *recommended*. |
70
- | [Mistral-Small-3.1-24B-Instruct-2503-Q5_K_M.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q5_K_M.gguf) | Q5_K_M | 16.76GB | false | High quality, *recommended*. |
71
- | [Mistral-Small-3.1-24B-Instruct-2503-Q5_K_S.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q5_K_S.gguf) | Q5_K_S | 16.30GB | false | High quality, *recommended*. |
72
  | [Mistral-Small-3.1-24B-Instruct-2503-Q5_K_S.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q5_K_S.gguf) | Q5_K_S | 16.30GB | false | High quality, *recommended*. |
73
  | [Mistral-Small-3.1-24B-Instruct-2503-Q4_1.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_1.gguf) | Q4_1 | 14.87GB | false | Legacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon. |
74
- | [Mistral-Small-3.1-24B-Instruct-2503-Q4_1.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_1.gguf) | Q4_1 | 14.87GB | false | Legacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon. |
75
  | [Mistral-Small-3.1-24B-Instruct-2503-Q4_K_L.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_L.gguf) | Q4_K_L | 14.83GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
76
- | [Mistral-Small-3.1-24B-Instruct-2503-Q4_K_L.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_L.gguf) | Q4_K_L | 14.83GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
77
- | [Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf) | Q4_K_M | 14.33GB | false | Good quality, default size for most use cases, *recommended*. |
78
  | [Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf) | Q4_K_M | 14.33GB | false | Good quality, default size for most use cases, *recommended*. |
79
  | [Mistral-Small-3.1-24B-Instruct-2503-Q4_K_S.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_S.gguf) | Q4_K_S | 13.55GB | false | Slightly lower quality with more space savings, *recommended*. |
80
- | [Mistral-Small-3.1-24B-Instruct-2503-Q4_K_S.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_S.gguf) | Q4_K_S | 13.55GB | false | Slightly lower quality with more space savings, *recommended*. |
81
- | [Mistral-Small-3.1-24B-Instruct-2503-Q4_0.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_0.gguf) | Q4_0 | 13.49GB | false | Legacy format, offers online repacking for ARM and AVX CPU inference. |
82
  | [Mistral-Small-3.1-24B-Instruct-2503-Q4_0.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_0.gguf) | Q4_0 | 13.49GB | false | Legacy format, offers online repacking for ARM and AVX CPU inference. |
83
  | [Mistral-Small-3.1-24B-Instruct-2503-IQ4_NL.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-IQ4_NL.gguf) | IQ4_NL | 13.47GB | false | Similar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference. |
84
- | [Mistral-Small-3.1-24B-Instruct-2503-IQ4_NL.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-IQ4_NL.gguf) | IQ4_NL | 13.47GB | false | Similar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference. |
85
- | [Mistral-Small-3.1-24B-Instruct-2503-Q3_K_XL.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q3_K_XL.gguf) | Q3_K_XL | 12.99GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
86
  | [Mistral-Small-3.1-24B-Instruct-2503-Q3_K_XL.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q3_K_XL.gguf) | Q3_K_XL | 12.99GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
87
  | [Mistral-Small-3.1-24B-Instruct-2503-IQ4_XS.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-IQ4_XS.gguf) | IQ4_XS | 12.76GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
88
- | [Mistral-Small-3.1-24B-Instruct-2503-IQ4_XS.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-IQ4_XS.gguf) | IQ4_XS | 12.76GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
89
  | [Mistral-Small-3.1-24B-Instruct-2503-Q3_K_L.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q3_K_L.gguf) | Q3_K_L | 12.40GB | false | Lower quality but usable, good for low RAM availability. |
90
  | [Mistral-Small-3.1-24B-Instruct-2503-Q3_K_M.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q3_K_M.gguf) | Q3_K_M | 11.47GB | false | Low quality. |
91
  | [Mistral-Small-3.1-24B-Instruct-2503-IQ3_M.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-IQ3_M.gguf) | IQ3_M | 10.65GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |
 
59
  | -------- | ---------- | --------- | ----- | ----------- |
60
  | [Mistral-Small-3.1-24B-Instruct-2503-bf16.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-bf16.gguf) | bf16 | 47.15GB | false | Full BF16 weights. |
61
  | [Mistral-Small-3.1-24B-Instruct-2503-Q8_0.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q8_0.gguf) | Q8_0 | 25.05GB | false | Extremely high quality, generally unneeded but max available quant. |
 
 
62
  | [Mistral-Small-3.1-24B-Instruct-2503-Q6_K_L.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q6_K_L.gguf) | Q6_K_L | 19.67GB | false | Uses Q8_0 for embed and output weights. Very high quality, near perfect, *recommended*. |
63
  | [Mistral-Small-3.1-24B-Instruct-2503-Q6_K.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q6_K.gguf) | Q6_K | 19.35GB | false | Very high quality, near perfect, *recommended*. |
 
 
64
  | [Mistral-Small-3.1-24B-Instruct-2503-Q5_K_L.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q5_K_L.gguf) | Q5_K_L | 17.18GB | false | Uses Q8_0 for embed and output weights. High quality, *recommended*. |
65
  | [Mistral-Small-3.1-24B-Instruct-2503-Q5_K_M.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q5_K_M.gguf) | Q5_K_M | 16.76GB | false | High quality, *recommended*. |
 
 
66
  | [Mistral-Small-3.1-24B-Instruct-2503-Q5_K_S.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q5_K_S.gguf) | Q5_K_S | 16.30GB | false | High quality, *recommended*. |
67
  | [Mistral-Small-3.1-24B-Instruct-2503-Q4_1.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_1.gguf) | Q4_1 | 14.87GB | false | Legacy format, similar performance to Q4_K_S but with improved tokens/watt on Apple silicon. |
 
68
  | [Mistral-Small-3.1-24B-Instruct-2503-Q4_K_L.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_L.gguf) | Q4_K_L | 14.83GB | false | Uses Q8_0 for embed and output weights. Good quality, *recommended*. |
 
 
69
  | [Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf) | Q4_K_M | 14.33GB | false | Good quality, default size for most use cases, *recommended*. |
70
  | [Mistral-Small-3.1-24B-Instruct-2503-Q4_K_S.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_S.gguf) | Q4_K_S | 13.55GB | false | Slightly lower quality with more space savings, *recommended*. |
 
 
71
  | [Mistral-Small-3.1-24B-Instruct-2503-Q4_0.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_0.gguf) | Q4_0 | 13.49GB | false | Legacy format, offers online repacking for ARM and AVX CPU inference. |
72
  | [Mistral-Small-3.1-24B-Instruct-2503-IQ4_NL.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-IQ4_NL.gguf) | IQ4_NL | 13.47GB | false | Similar to IQ4_XS, but slightly larger. Offers online repacking for ARM CPU inference. |
 
 
73
  | [Mistral-Small-3.1-24B-Instruct-2503-Q3_K_XL.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q3_K_XL.gguf) | Q3_K_XL | 12.99GB | false | Uses Q8_0 for embed and output weights. Lower quality but usable, good for low RAM availability. |
74
  | [Mistral-Small-3.1-24B-Instruct-2503-IQ4_XS.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-IQ4_XS.gguf) | IQ4_XS | 12.76GB | false | Decent quality, smaller than Q4_K_S with similar performance, *recommended*. |
 
75
  | [Mistral-Small-3.1-24B-Instruct-2503-Q3_K_L.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q3_K_L.gguf) | Q3_K_L | 12.40GB | false | Lower quality but usable, good for low RAM availability. |
76
  | [Mistral-Small-3.1-24B-Instruct-2503-Q3_K_M.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q3_K_M.gguf) | Q3_K_M | 11.47GB | false | Low quality. |
77
  | [Mistral-Small-3.1-24B-Instruct-2503-IQ3_M.gguf](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/blob/main/mistralai_Mistral-Small-3.1-24B-Instruct-2503-IQ3_M.gguf) | IQ3_M | 10.65GB | false | Medium-low quality, new method with decent performance comparable to Q3_K_M. |