|
--- |
|
library_name: transformers |
|
base_model: |
|
- deepseek-ai/DeepSeek-V2-Lite |
|
--- |
|
|
|
# ⚠️ Note: |
|
|
|
These model weights are for personal testing purposes only. The goal is to find a quantization method that achieves high compression while preserving as much of the model's original performance as possible. The current compression scheme may not be optimal, so please use these weights with caution. |
|
|
|
# Creation |
|
This model was created by applying the ```bitsandbytes``` and ```transformers``` as presented in the code snipet below. |
|
|
|
```Python |
|
import torch |
|
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig |
|
|
|
model_name = "deepseek-ai/DeepSeek-V2-Lite" |
|
bnb_config = BitsAndBytesConfig( |
|
load_in_4bit=True, |
|
bnb_4bit_quant_type="nf4", # NF4 for weight |
|
bnb_4bit_use_double_quant=True, |
|
bnb_4bit_compute_dtype=torch.bfloat16 # bnb supports bfloat16 |
|
) |
|
|
|
bnb_model = AutoModelForCausalLM.from_pretrained( |
|
model_name, |
|
trust_remote_code=True, |
|
torch_dtype=torch.bfloat16, |
|
quantization_config=bnb_config |
|
) |
|
|
|
tokenizer = AutoTokenizer.from_pretrained( model_name, trust_remote_code=True) |
|
|
|
bnb_model.push_to_hub("basicv8vc/DeepSeek-V2-Lite-bnb-4bit") |
|
tokenizer.push_to_hub("basicv8vc/DeepSeek-V2-Lite-bnb-4bit") |
|
|
|
|
|
``` |
|
|
|
|
|
# Sources |
|
|
|
https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite |
