|
|
--- |
|
|
license: other |
|
|
datasets: |
|
|
- lemonilia/LimaRP |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- mistralai/Mixtral-8x7B-Instruct-v0.1 |
|
|
- Doctor-Shotgun/Mixtral-8x7B-Instruct-v0.1-LimaRP-ZLoss |
|
|
--- |
|
|
|
|
|
This is Doctor-Shotgun's LimaRP-zloss fine tune applied directly to mistralai/Mixtral-8x7B-Instruct-v0.1, as opposed to the base model from which the qLORA was trained (Mixtral-8X7B-v0.1). Traditionally a tune is applied to the base model then merged to Instruct); we don't do that here. We call these "uphill" models since the qLORA is pushed up the hill without an intermediate merge. This creates a more stable model which can be used as part of a merge of Instruct-based models. |
|
|
|
|
|
Our use case for this is to apply a very minimal density and weight of LimaRP (which has lots of model-ism pseudo-slop) as part of a more nuanced, more instruction-following merge with "uphill" variants of our proprietary fine tunes (crush/crest/clash) |
