|
|
--- |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- togethercomputer/RedPajama-Data-1T |
|
|
- togethercomputer/RedPajama-Data-Instruct |
|
|
- EleutherAI/pile |
|
|
language: |
|
|
- en |
|
|
library_name: transformers |
|
|
--- |
|
|
|
|
|
# Llama-2-7B-32KCtx |
|
|
|
|
|
|
|
|
# Install Flash Attention For Inference with 32K |
|
|
|
|
|
``` |
|
|
export CUDA_HOME=/usr/local/cuda-11.8 |
|
|
pip install ninja |
|
|
pip install flash-attn --no-build-isolation |
|
|
pip install git+https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary |
|
|
``` |
|
|
Please revise the path of `CUDA_HOME`. `ninja` is needed to accelerate the process of compiling. |
|
|
|
|
|
And then: |
|
|
```python |
|
|
model = AutoModelForCausalLM.from_pretrained('togethercomputer/Llama-2-7B-32KCtx-v0.1', trust_remote_code=True, torch_dtype=torch.float16) |
|
|
``` |
|
|
|
|
|
You can also use vanilla `transformers` to load this model: |
|
|
```python |
|
|
model = AutoModelForCausalLM.from_pretrained('togethercomputer/Llama-2-7B-32KCtx-v0.1', torch_dtype=torch.float16) |
|
|
``` |
|
|
|
|
|
|
|
|
TODO |
