Hugging Face's logo

mradermacher
/
DeepHermes-3-Mistral-24B-Preview-i1-GGUF

Transformers
GGUF
English
Mistral-Small
instruct
finetune
chatml
gpt4
synthetic data
distillation
function calling
json mode
axolotl
roleplaying
chat
reasoning
r1
vllm
Inference Endpoints
imatrix
conversational
Model card Files Community
mradermacher commited on
Commit
4d1a5b1
·
verified ·
1 Parent(s): 1e59709

auto-patch README.md

Browse files
Files changed (1) hide show
  1. README.md +85 -0
README.md CHANGED
@@ -1,6 +1,91 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  <!-- ### quantize_version: 2 -->
2
  <!-- ### output_tensor_quantised: 1 -->
3
  <!-- ### convert_type: hf -->
4
  <!-- ### vocab_type: -->
5
  <!-- ### tags: nicoboss -->
6
  weighted/imatrix quants of https://huggingface.co/NousResearch/DeepHermes-3-Mistral-24B-Preview
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: NousResearch/DeepHermes-3-Mistral-24B-Preview
3
+ language:
4
+ - en
5
+ library_name: transformers
6
+ license: apache-2.0
7
+ quantized_by: mradermacher
8
+ tags:
9
+ - Mistral-Small
10
+ - instruct
11
+ - finetune
12
+ - chatml
13
+ - gpt4
14
+ - synthetic data
15
+ - distillation
16
+ - function calling
17
+ - json mode
18
+ - axolotl
19
+ - roleplaying
20
+ - chat
21
+ - reasoning
22
+ - r1
23
+ - vllm
24
+ ---
25
+ ## About
26
+
27
  <!-- ### quantize_version: 2 -->
28
  <!-- ### output_tensor_quantised: 1 -->
29
  <!-- ### convert_type: hf -->
30
  <!-- ### vocab_type: -->
31
  <!-- ### tags: nicoboss -->
32
  weighted/imatrix quants of https://huggingface.co/NousResearch/DeepHermes-3-Mistral-24B-Preview
33
+
34
+ <!-- provided-files -->
35
+ static quants are available at https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-GGUF
36
+ ## Usage
37
+
38
+ If you are unsure how to use GGUF files, refer to one of [TheBloke's
39
+ READMEs](https://huggingface.co/TheBloke/KafkaLM-70B-German-V0.1-GGUF) for
40
+ more details, including on how to concatenate multi-part files.
41
+
42
+ ## Provided Quants
43
+
44
+ (sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants)
45
+
46
+ | Link | Type | Size/GB | Notes |
47
+ |:-----|:-----|--------:|:------|
48
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-IQ1_S.gguf) | i1-IQ1_S | 5.4 | for the desperate |
49
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-IQ1_M.gguf) | i1-IQ1_M | 5.9 | mostly desperate |
50
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-IQ2_XXS.gguf) | i1-IQ2_XXS | 6.6 | |
51
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-IQ2_XS.gguf) | i1-IQ2_XS | 7.3 | |
52
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-IQ2_S.gguf) | i1-IQ2_S | 7.6 | |
53
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-IQ2_M.gguf) | i1-IQ2_M | 8.2 | |
54
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-Q2_K_S.gguf) | i1-Q2_K_S | 8.4 | very low quality |
55
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-Q2_K.gguf) | i1-Q2_K | 9.0 | IQ3_XXS probably better |
56
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-IQ3_XXS.gguf) | i1-IQ3_XXS | 9.4 | lower quality |
57
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-IQ3_XS.gguf) | i1-IQ3_XS | 10.0 | |
58
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-Q3_K_S.gguf) | i1-Q3_K_S | 10.5 | IQ3_XS probably better |
59
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-IQ3_S.gguf) | i1-IQ3_S | 10.5 | beats Q3_K* |
60
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-IQ3_M.gguf) | i1-IQ3_M | 10.8 | |
61
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-Q3_K_M.gguf) | i1-Q3_K_M | 11.6 | IQ3_S probably better |
62
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-Q3_K_L.gguf) | i1-Q3_K_L | 12.5 | IQ3_M probably better |
63
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-IQ4_XS.gguf) | i1-IQ4_XS | 12.9 | |
64
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-Q4_0.gguf) | i1-Q4_0 | 13.6 | fast, low quality |
65
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-Q4_K_S.gguf) | i1-Q4_K_S | 13.6 | optimal size/speed/quality |
66
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-Q4_K_M.gguf) | i1-Q4_K_M | 14.4 | fast, recommended |
67
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-Q4_1.gguf) | i1-Q4_1 | 15.0 | |
68
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-Q5_K_S.gguf) | i1-Q5_K_S | 16.4 | |
69
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-Q5_K_M.gguf) | i1-Q5_K_M | 16.9 | |
70
+ | [GGUF](https://huggingface.co/mradermacher/DeepHermes-3-Mistral-24B-Preview-i1-GGUF/resolve/main/DeepHermes-3-Mistral-24B-Preview.i1-Q6_K.gguf) | i1-Q6_K | 19.4 | practically like static Q6_K |
71
+
72
+ Here is a handy graph by ikawrakow comparing some lower-quality quant
73
+ types (lower is better):
74
+
75
+ ![image.png](https://www.nethype.de/huggingface_embed/quantpplgraph.png)
76
+
77
+ And here are Artefact2's thoughts on the matter:
78
+ https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9
79
+
80
+ ## FAQ / Model Request
81
+
82
+ See https://huggingface.co/mradermacher/model_requests for some answers to
83
+ questions you might have and/or if you want some other model quantized.
84
+
85
+ ## Thanks
86
+
87
+ I thank my company, [nethype GmbH](https://www.nethype.de/), for letting
88
+ me use its servers and providing upgrades to my workstation to enable
89
+ this work in my free time. Additional thanks to [@nicoboss](https://huggingface.co/nicoboss) for giving me access to his private supercomputer, enabling me to provide many more imatrix quants, at much higher quality, than I would otherwise be able to.
90
+
91
+ <!-- end -->