davzoku/moecule-3x1b-m6-fks

🫐 Moecule 3x1B M6 FKS

Model Details

This model is a mixture of experts (MoE) using the RhuiDih/moetify library with various task-specific experts. All relevant expert models, LoRA adapters, and datasets are available at Moecule Ingredients.

Key Features

• Zero Additional Training: Combine existing domain-specific / task-specific experts into a powerful MoE model without additional training!

System Requirements

Steps	System Requirements
MoE Creation	> 25.3 GB System RAM
Inference (fp16)	GPU with > 7GB VRAM

MoE Creation

To reproduce this model, run the following command:

# git clone moetify fork that fixes dependency issue
!git clone -b fix-transformers-4.47.1-FlashA2-dependency --single-branch https://github.com/davzoku/moetify.git

!cd moetify && pip install -e .

python -m moetify.mix \
 --output_dir ./moecule-3x1b-m6-fks \
 --model_path unsloth/llama-3.2-1b-Instruct \
 --modules mlp q_proj \
 --ingredients \
 davzoku/finqa_expert_1b \
 davzoku/kyc_expert_1b \
 davzoku/stock_market_expert_1b

Model Parameters

INFO:root:Stem parameters: 626067456
INFO:root:Experts parameters: 2617245696
INFO:root:Routers parameters: 196608
INFO:root:MOE total parameters (numel): 3243509760
INFO:root:MOE total parameters : 3243509760
INFO:root:MOE active parameters: 2371094528

Inference

To run an inference with this model, you can use the following code snippet:

# git clone moetify fork that fixes dependency issue
!git clone -b fix-transformers-4.47.1-FlashA2-dependency --single-branch https://github.com/davzoku/moetify.git

!cd moetify && pip install -e .

model = AutoModelForCausalLM.from_pretrained(<model-name>, device_map='auto', trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(<model-name>)

def format_instruction(row):
    return f"""### Question: {row}"""

greedy_generation_config = GenerationConfig(
    temperature=0.1,
    top_p=0.75,
    top_k=40,
    num_beams=1,
    max_new_tokens=128,
    repetition_penalty=1.2
)


input_text = "In what ways did Siemens's debt restructuring on March 06, 2024 reflect its strategic priorities?"
formatted_input = format_instruction(input_text)
inputs = tokenizer(formatted_input, return_tensors="pt").to('cuda')

with torch.no_grad():
    outputs = model.generate(
        input_ids=inputs.input_ids,
        attention_mask=inputs.attention_mask,
        generation_config=greedy_generation_config
    )

generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

The Team

• CHOCK Wan Kee
• Farlin Deva Binusha DEVASUGIN MERLISUGITHA
• GOH Bao Sheng
• Jessica LEK Si Jia
• Sinha KHUSHI
• TENG Kok Wai (Walter)

References

• Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts
• RhuiDih/moetify