Create README.md

README.md

@@ -0,0 +1,27 @@
+---
+license: apache-2.0
+library_name: vllm
+pipeline_tag: image-text-to-text
+tags:
+- int4
+- vllm
+- llmcompressor
+base_model: mistralai/Mistral-Small-3.1-24B-Instruct-2503
+---
+
+# Mistral-Small-3.1-24B-Instruct-2503-GPTQ-4b-128g
+
+## Model Overview
+
+This model was obtained by quantizing the weights of [Mistral-Small-3.1-24B-Instruct-2503](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503) to INT4 data type. This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%.
+
+Only the weights of the linear operators within `language_model` transformers blocks are quantized. Vision model and multimodal projection are kept in original precision. Weights are quantized using a symmetric per-group scheme, with group size 128. The GPTQ algorithm is applied for quantization.
+
+Model checkpoint is saved in [compressed_tensors](https://github.com/neuralmagic/compressed-tensors) format.
+
+## Usage 
+
+* To use the model in `transformers` update the package to stable release of Mistral-3
+  
+  `pip install git+https://github.com/huggingface/[email protected]`
+* To use the model in `vLLM` update the package to version `vllm>=0.8.0`.