|
--- |
|
base_model: |
|
- mistralai/Mistral-Small-3.1-24B-Instruct-2503 |
|
--- |
|
|
|
Text-Only EXL2 Quant of [mistralai/Mistral-Small-3.1-24B-Instruct-2503](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503) |
|
|
|
The following changes were made: |
|
|
|
1. Vision encoder removed |
|
2. Architecture changed to that of [mistralai/Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501) |
|
3. Chat template in tokenizer_config.json was modified (see below). |
|
|
|
I was having trouble with the timestamp at the beginning of the system prompt and removed it from tokenizer_config.json. |
|
|
|
**NOTE** Tensor Parallel is not implemented in exllamav2 for both [mistralai/Mistral-Small-3.1-24B-Instruct-2503](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503) and [mistralai/Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501). |
