NewstaR/Porpoise-6b-instruct

This model is a finetuned version of the DeciLM-6b-instruct on the Dolphin GPT4 Dataset

Please set naive_attention_prefill to true when loading this model.

Example:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, AutoTokenizer

model_name = "NewstaR/Porpoise-6b-instruct"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    trust_remote_code=True,
    naive_attention_prefill=True,
)
model.config.use_cache = False