Bielik-11B-v2-bnb-4bit

BitsAndBytes 4bit quantization

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# Configure quantization
quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype="float16",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4"
)

# Load the model with quantization
model_name = "speakleash/Bielik-11B-v2.3-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=quant_config,
    device_map="auto"
)

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Save the model and tokenizer
save_directory = "./quantized_model"
model.save_pretrained(save_directory)
tokenizer.save_pretrained(save_directory)

Base Model description:

Developed by: SpeakLeash & ACK Cyfronet AGH
Language: Polish
Model type: causal decoder-only
Initialized from: Mistral-7B-v0.2
License: Apache 2.0 and Terms of Use
Model ref: speakleash:45b6efdb701991181a05968fc53d2a8e

Limitations and Biases

Bielik-11B-v2 is not intended for deployment without fine-tuning. It should not be used for human-facing interactions without further guardrails and user consent.

Bielik-11B-v2 can produce factually incorrect output, and should not be relied on to produce factually accurate data. Bielik-11B-v2 was trained on various public datasets. While great efforts have been taken to clear the training data, it is possible that this model can generate lewd, false, biased or otherwise offensive outputs.

Citation

@misc{Bielik11Bv2b,
    title     = {Bielik-11B-v2 model card},
    author    = {Ociepa, Krzysztof and Flis, Łukasz and Wróbel, Krzysztof and Gwoździej, Adrian and {SpeakLeash Team} and {Cyfronet Team}},
    year      = {2024},
    url       = {https://huggingface.co/speakleash/Bielik-11B-v2},
    note      = {Accessed: 2024-08-28},
    urldate   = {2024-08-28}
}
@unpublished{Bielik11Bv2a,
  author = {Ociepa, Krzysztof and Flis, Łukasz and Kinas, Remigiusz and Gwoździej, Adrian and Wróbel, Krzysztof},
  title  = {Bielik: A Family of Large Language Models for the Polish Language - Development, Insights, and Evaluation},
  year   = {2024},
}
@misc{ociepa2024bielik7bv01polish,
      title={Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation}, 
      author={Krzysztof Ociepa and Łukasz Flis and Krzysztof Wróbel and Adrian Gwoździej and Remigiusz Kinas},
      year={2024},
      eprint={2410.18565},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.18565}, 
}