File size: 909 Bytes
28205cd 89a7e75 28205cd 89a7e75 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
---
license: apache-2.0
datasets:
- togethercomputer/RedPajama-Data-1T
- togethercomputer/RedPajama-Data-Instruct
- EleutherAI/pile
language:
- en
library_name: transformers
---
# Llama-2-7B-32KCtx
# Install Flash Attention For Inference with 32K
```
export CUDA_HOME=/usr/local/cuda-11.8
pip install ninja
pip install flash-attn --no-build-isolation
pip install git+https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary
```
Please revise the path of `CUDA_HOME`. `ninja` is needed to accelerate the process of compiling.
And then:
```python
model = AutoModelForCausalLM.from_pretrained('togethercomputer/Llama-2-7B-32KCtx-v0.1', trust_remote_code=True, torch_dtype=torch.float16)
```
You can also use vanilla `transformers` to load this model:
```python
model = AutoModelForCausalLM.from_pretrained('togethercomputer/Llama-2-7B-32KCtx-v0.1', torch_dtype=torch.float16)
```
TODO |