basicv8vc/Moonlight-16B-A3B-bnb-4bit

⚠️ Note:

These model weights are for personal testing purposes only. The goal is to find a quantization method that achieves high compression while preserving as much of the model's original performance as possible. The current compression scheme may not be optimal, so please use these weights with caution.

Creation

This model was created by applying the bitsandbytes and transformers as presented in the code snipet below.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

model_name = "moonshotai/Moonlight-16B-A3B"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",  # NF4 for weight
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16  # bnb supports bfloat16
)

bnb_model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    quantization_config=bnb_config
)

tokenizer = AutoTokenizer.from_pretrained( model_name, trust_remote_code=True)

bnb_model.push_to_hub("basicv8vc/Moonlight-16B-A3B-bnb-4bit")
tokenizer.push_to_hub("basicv8vc/Moonlight-16B-A3B-bnb-4bit")

Sources

https://huggingface.co/moonshotai/Moonlight-16B-A3B