Update README.md
Browse files
README.md
CHANGED
@@ -16,7 +16,7 @@ base_model:
|
|
16 |
- mistralai/Mistral-Small-24B-Instruct-2501
|
17 |
pipeline_tag: text-classification
|
18 |
---
|
19 |
-
# 🔥 Quantized Model: Mistral-Small-24B-Instruct-
|
20 |
|
21 |
This is a 4-bit quantized version of [mistralai/Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501) model, quantized by [ConfidentialMind.com](https://www.confidentialmind.com) 🤖✨
|
22 |
It leverages the open-source GPTQModel quantization to achieve 4-bit precision with a group size of 128 resulting in a
|
@@ -26,6 +26,7 @@ faster model with minimal performance degradation.
|
|
26 |
Ran on a single NVIDIA A100 GPU with 80GB of VRAM.
|
27 |
|
28 |
*Note* `batch_size` is set quite high as the model is small, you may need to adjust this to your GPU VRAM.
|
|
|
29 |
|
30 |
## Model Details
|
31 |
- **Original Model:** [mistralai/Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501)
|
|
|
16 |
- mistralai/Mistral-Small-24B-Instruct-2501
|
17 |
pipeline_tag: text-classification
|
18 |
---
|
19 |
+
# 🔥 Quantized Model: Mistral-Small-24B-Instruct-2501_GPTQ_G128_W4A16_MSE 🔥
|
20 |
|
21 |
This is a 4-bit quantized version of [mistralai/Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501) model, quantized by [ConfidentialMind.com](https://www.confidentialmind.com) 🤖✨
|
22 |
It leverages the open-source GPTQModel quantization to achieve 4-bit precision with a group size of 128 resulting in a
|
|
|
26 |
Ran on a single NVIDIA A100 GPU with 80GB of VRAM.
|
27 |
|
28 |
*Note* `batch_size` is set quite high as the model is small, you may need to adjust this to your GPU VRAM.
|
29 |
+
*Note2* Due to the "packed" nature of mistral-small weights, MSE was used agressively along with a higher damping factor - this resulted in lesser loss and perplexity, however G32 is more advised
|
30 |
|
31 |
## Model Details
|
32 |
- **Original Model:** [mistralai/Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501)
|