JustJaro commited on
Commit
8033938
·
verified ·
1 Parent(s): 87e6612

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -1
README.md CHANGED
@@ -16,7 +16,7 @@ base_model:
16
  - mistralai/Mistral-Small-24B-Instruct-2501
17
  pipeline_tag: text-classification
18
  ---
19
- # 🔥 Quantized Model: Mistral-Small-24B-Instruct-2501_gptq_g128_4bit 🔥
20
 
21
  This is a 4-bit quantized version of [mistralai/Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501) model, quantized by [ConfidentialMind.com](https://www.confidentialmind.com) 🤖✨
22
  It leverages the open-source GPTQModel quantization to achieve 4-bit precision with a group size of 128 resulting in a
@@ -26,6 +26,7 @@ faster model with minimal performance degradation.
26
  Ran on a single NVIDIA A100 GPU with 80GB of VRAM.
27
 
28
  *Note* `batch_size` is set quite high as the model is small, you may need to adjust this to your GPU VRAM.
 
29
 
30
  ## Model Details
31
  - **Original Model:** [mistralai/Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501)
 
16
  - mistralai/Mistral-Small-24B-Instruct-2501
17
  pipeline_tag: text-classification
18
  ---
19
+ # 🔥 Quantized Model: Mistral-Small-24B-Instruct-2501_GPTQ_G128_W4A16_MSE 🔥
20
 
21
  This is a 4-bit quantized version of [mistralai/Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501) model, quantized by [ConfidentialMind.com](https://www.confidentialmind.com) 🤖✨
22
  It leverages the open-source GPTQModel quantization to achieve 4-bit precision with a group size of 128 resulting in a
 
26
  Ran on a single NVIDIA A100 GPU with 80GB of VRAM.
27
 
28
  *Note* `batch_size` is set quite high as the model is small, you may need to adjust this to your GPU VRAM.
29
+ *Note2* Due to the "packed" nature of mistral-small weights, MSE was used agressively along with a higher damping factor - this resulted in lesser loss and perplexity, however G32 is more advised
30
 
31
  ## Model Details
32
  - **Original Model:** [mistralai/Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501)