| license: apache-2.0 | |
| # Mixtral-8x7B--v0.1: Model 7 | |
| ## Model Description | |
| This model is the 7th extracted standalone model from the [mistralai/Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1), using the [Mixtral Model Expert Extractor tool](https://github.com/MeNicefellow/Mixtral-Model-Expert-Extractor) I made. It is constructed by selecting the first expert from each Mixture of Experts (MoE) layer. The extraction of this model is experimental. It is expected to be worse than Mistral-7B. | |
| ## Model Architecture | |
| The architecture of this model includes: | |
| - Multi-head attention layers derived from the base Mixtral model. | |
| - The first expert from each MoE layer, intended to provide a balanced approach to language understanding and generation tasks. | |
| - Additional layers and components as required to ensure the model's functionality outside the MoE framework. | |
| ### Example | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| model_name = "DrNicefellow/Mistral-3-from-Mixtral-8x7B-v0.1" | |
| tokenizer = AutoTokenizer.from_pretrained(model_name) | |
| model = AutoModelForCausalLM.from_pretrained(model_name) | |
| text = "Today is a pleasant" | |
| input_ids = tokenizer.encode(text, return_tensors='pt') | |
| output = model.generate(input_ids) | |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) | |
| ``` | |
| ## License | |
| This model is available under the Apache 2.0 License. | |
| ## Discord Server | |
| Join our Discord server [here](https://discord.gg/xhcBDEM3). | |
| ## License | |
| This model is open-sourced under the Apache 2.0 License. See the LICENSE file for more details. | |