TMElyralab/lyraLLaMA · Hugging Face

Special Notes

Due to the license regulation of LLaMA, we are not allowed to release the accelerated parameters directly. We hope to discuss with you guys to figure out a legal way to share lyraLLaMA. If you have any suggestions, please feel free to drop us a line at [email protected].

Model Card for lyraLLaMA

lyraLLaMA is currently the fastest LLaMA-13b available. The inference speed of lyraLLaMA has achieved 3000+ tokens/s on A100, up to 6x acceleration upon the torch version.

Among its main features are:

device: Nvidia GPU with Amperer architecture or Volta architecture (A100 or higher, V100).
batch_size: compiled with dynamic batch size, maximum depends on device.

We use the LLaMA.13B model for measurement, but this optimized inference is applicable to LLaMA models of different sizes.

Speed

Evaluated at tokens/s
test on A100 40G
fp16 and MEMOPT precision

LLaMA-Ziya-13B

Version	Batch Size 1	Batch Size 8	Batch Size 16	Batch Size 32	Batch Size 64
Torch LLaMA	31.74	289.2	521.37	775.69	OOM
lyraLLaMA fp16	73.2	565.6	1179.59	1795.63	3061.27
lyraLLaMA MEMOPT	104	770.5	1389.9	2390.4	3782.1

LLaMA-Vicuna-13B

Version	Batch Size 1	Batch Size 8	Batch Size 16	Batch Size 32	Batch Size 64
Torch LLaMA	24.65	167.3	322.97	407.99	OOM
lyraLLaMA fp16	53.67	421.38	804.31	1519.28	2679.82
lyraLLaMA MEMOPT	79.81	603.15	1117.27	1966.52	3200.32

Docker Environment Recommendation

For Cuda 11.X: we recommend nvcr.io/nvidia/pytorch:22.12-py3
For Cuda 12.0: we recommend nvcr.io/nvidia/pytorch:23.02-py3

docker pull nvcr.io/nvidia/pytorch:23.02-py3
docker run --rm -it --gpus all -v ./:/lyraLLaMA nvcr.io/nvidia/pytorch:23.02-py3

pip install -r requirements.txt
python demo.py

Uses

from lyra_llama import lyraLLaMA

model_path = "./models/lamma-13b-1-gpu-fp16.bin"
tokenizer_path = "./models/"
dtype='fp16'
prompt = "今天天气大概 25度，有点小雨，吹着风，我想去户外散步，应该穿什么样的衣服 裤子鞋子搭配"
max_output_length = 512
memopt_mode = 0 # To use MEMOPT mode, set memopt_mode=1

model = lyraLLaMA(model_path, tokenizer_path, dtype, memopt_mode)

prompt = '<human>:' + prompt.strip() + '\n<bot>:'

bs = 1
prompts = [prompt, ] * bs
output_texts = model.generate(
        prompts, output_length=max_output_length,
        top_k=30, top_p=0.85, temperature=1.0, repetition_penalty=1.0, do_sample=False)

print(output_texts)

Demo Outputs

LLaMA-Ziya-13B

input

今天天气大概 25度，有点小雨，吹着风，我想去户外散步，应该穿什么样的衣服裤子鞋子搭配

output

今天天气大概25度，有点小雨，吹着风。建议穿上轻便的运动服装，比如：

上衣：选择透气性好的运动T恤或短袖运动衫，颜色可以选择暗色系，以免晒伤。
裤子：可以选择运动裤或者休闲裤，最好选择能够吸湿排汗的材质，以便于散步时舒适。
鞋子：建议穿上运动鞋或者轻便的休闲鞋，以便于走路。鞋底需要有良好的防滑性，以防止滑倒。
外套：如果雨比较大，可以搭配一件轻便的雨衣或者防雨衣，以保护身体不受雨淋。总之，建议穿上轻便、舒适的运动服装，搭配一双合适的鞋子和外套，以便于户外散步。

LLaMA-Vicuna-13B

input

List 3 different machine learning algorithms and explain where they are applicable.

output

Outputs:

Linear Regression: Linear regression is a simple, yet powerful algorithm that is used to predict the value of a continuous variable based on one or more input features. It is commonly used in prediction and forecasting tasks, such as predicting the price of a stock or the sales of a product.
Decision Trees: Decision Trees are a type of supervised learning algorithm that can be used for both classification and regression tasks. They work by partitioning the feature space into smaller subspaces, with each subspace corresponding to a leaf node in the tree. Decision Trees are commonly used in applications such as credit risk assessment and customer segmentation.
Support Vector Machines (SVMs): SVMs are a type of supervised learning algorithm that can be used for both classification and regression tasks. They work by finding the best hyperplane that separates the data into different classes. SVMs are commonly used in applications such as image classification and natural language processing.

TODO

Support for int4
Inference for longer context situations
Streaming inference mode.

Citation

@Misc{lyraLLaMA2023,
  author =       {Kangjian Wu, Zhengtao Wang, Yibo Lu, Bin Wu},
  title =        {lyraLLaMA: Accelerating LLaMA-13b(fp16) to 3000+ tokens/s},
  howpublished = {\url{https://huggingface.co/TMElyralab/lyraLLaMA}},
  year =         {2023}
}

Report bug

start a discussion to report any bugs!--> https://huggingface.co/TMElyralab/lyraLLaMA/discussions
report bug with a [bug] mark in the title.