Hugging Face's logo

khoicrtp
/
test_qlora

TensorBoard
Model card Files Metrics Community
test_qlora
File size: 5,147 Bytes 
947b9bd
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
 
# LLM Fine-Tuning with QLoRA

This repository can help to instruct-tune Open LLaMA, RedPajama or StableLM models on consumer hardware using QLoRA (Original implementation [here](https://github.com/artidoro/qlora)). It's mostly based on the original alpaca-lora repo which can be found [here](https://github.com/tloen/alpaca-lora). Please note that this has only been tested on Open LLama 3b and RedPajama 3b Models, but should work with other models. Contributions are welcome!

### Local Setup

1. Install dependencies

   ```bash
   pip install -r requirements.txt
   ```

1. If bitsandbytes doesn't work, [install it from source.](https://github.com/TimDettmers/bitsandbytes/blob/main/compile_from_source.md) Windows users can follow [these instructions](https://github.com/tloen/alpaca-lora/issues/17).

## Training (finetune.py)

This file contains a straightforward application of QLoRA PEFT to the Open LLaMA / RedPajama / StableLM model, as well as some code related to prompt construction and tokenization. PRs adapting this code to support larger models are always welcome.

**Example usage:**

For Open LLaMa

    python finetune.py \
        --base_model 'openlm-research/open_llama_3b_600bt_preview' \
        --data_path '../datasets/dolly.json' \
        --num_epochs=3 \
        --cutoff_len=512 \
        --group_by_length \
        --output_dir='./dolly-lora-3b' \
        --lora_r=16 \
        --lora_target_modules='[q_proj,v_proj]'

For RedPajama

    python finetune.py   \
    --base_model='togethercomputer/RedPajama-INCITE-Base-3B-v1' \
    --data_path='../datasets/dolly.json'   \
    --num_epochs=3   \
    --cutoff_len=512   \
    --group_by_length   \
    --output_dir='./dolly-lora-rp-3b-t1' \
    --lora_r=16 \
    --lora_target_modules='["query_key_value"]' 
    
For StableLM

    python finetune.py  \
    --base_model='stabilityai/stablelm-base-alpha-3b' \
    --data_path='../datasets/dolly.json' \
    --num_epochs=3 \
    --cutoff_len=512 \
    --group_by_length  \
    --output_dir='./dolly-lora-st-3b-t1' \
    --lora_r=16 \
    --lora_target_modules='["query_key_value"]'
    
For Pythia

    python finetune.py  \
    --base_model='EleutherAI/pythia-6.9b-deduped' \
    --data_path='../datasets/dolly.json' \
    --num_epochs=1 \
    --cutoff_len=512 \
    --group_by_length  \
    --output_dir='./dolly-lora-pyt-6b-t1' \
    --lora_r=8 \
    --lora_target_modules='["query_key_value"]'

We can also tweak our hyperparameters (similar to alpaca-lora):

    python finetune.py \
        --base_model 'openlm-research/open_llama_3b_600bt_preview \
        --data_path 'yahma/alpaca-cleaned' \
        --output_dir './lora-alpaca' \
        --batch_size 128 \
        --micro_batch_size 4 \
        --num_epochs 3 \
        --learning_rate 1e-4 \
        --cutoff_len 512 \
        --val_set_size 2000 \
        --lora_r 8 \
        --lora_alpha 16 \
        --lora_dropout 0.05 \
        --lora_target_modules '[q_proj,v_proj]' \
        --train_on_inputs \
        --group_by_length

## Inference (generate.py)
This file reads the foundation model from the Hugging Face model hub and the LoRA weights from trained peft model, and runs a Gradio interface for inference on a specified input. Users should treat this as example code for the use of the model, and modify it as needed.

Example usage:    

For Open LLaMa

    python generate.py \
        --base_model 'openlm-research/open_llama_3b_600bt_preview' \
        --lora_weights './lora-alpaca'
        
For RedPajama

    python generate.py  \
    --base_model 'togethercomputer/RedPajama-INCITE-Base-3B-v1'  \
    --lora_weights './dolly-lora-rp-3b-t1/'
       
For StableLM

    python generate.py  \
    --base_model 'stabilityai/stablelm-base-alpha-3b' \
    --lora_weights './dolly-lora-st-3b-t1'
    
For Pythia
   
    python generate.py  \
    --base_model 'EleutherAI/pythia-6.9b-deduped'  \
    --lora_weights './dolly-lora-pyt-6b-t1'
    
# Acknowledgements

We would like to express our heartfelt gratitude to **Meta** for releasing LLaMA . Without this pioneering technology, the foundations of projects like **Open Llama** and **Alpaca** wouldn't exist. We sincerely appreciate the immense contributions you've made to the field.

Our acknowledgements also extend to the teams behind **Open LLaMA**, **Together Computer**, **Alpaca** and **Alpaca LoRA**.. You can find more about their excellent work on their respective GitHub repositories:

- [Open Llama](https://github.com/openlm-research/open_llama)
- [Together Computer](https://github.com/togethercomputer)
- [Alpaca](https://github.com/tatsu-lab/stanford_alpaca)
- [Alpaca LoRa](https://github.com/tloen/alpaca-lora)

Lastly, we would like to express our thanks to the developers of **QLoRA** and **bitsandbytes** Your efforts have been instrumental in advancing the field, and we're grateful for your contributions. More information about these projects can be found at:

- [QLoRA](https://github.com/artidoro/qlora)
- [bitsandbytes](https://github.com/TimDettmers/bitsandbytes)


Thank you all for your commitment to innovation and for making these projects possible.