| license: cc-by-nc-2.0 | |
| by David, Fernando and Eric | |
| An experimentation regarding 'lasering' each expert to denoise and enhance model capabilities. | |
| This model has half size in comparison to the Mixtral 8x7b Instruct. And it basically has the same level of performance (we are working to get a better MMLU score). | |
| Used models (all lasered using laserRMT, except for the base model): | |
| *mlabonne/Marcoro14-7B-slerp (base) | |
| *cognitivecomputations/dolphin-2.6-mistral-7b-dpo | |
| *beowolx/CodeNinja-1.0-OpenChat-7B | |
| *Q-bert/MetaMath-Cybertron-Starling | |
| *WizardLM/WizardMath-7B-V1.1 | |
| It follows the implementation of laserRMT @ https://github.com/cognitivecomputations/laserRMT | |
| Here, we are controlling layers checking which ones have lower signal to noise ratios (which are more subject to noise), to apply Laser interventions, still using Machenko Pastur to calculate this ratio. | |
| We intend to be the first of a family of experimentations being carried out @ Cognitive Computations. | |
| In this experiment we have observed very high truthfulness and high reasoning capabilities. | |