|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- zh |
|
|
- en |
|
|
base_model: |
|
|
- Qwen/Qwen2.5-1.5B-Instruct-GPTQ-INT8 |
|
|
- Qwen/Qwen2.5-1.5B-Instruct-GPTQ-INT4 |
|
|
pipeline_tag: text-generation |
|
|
library_name: transformers |
|
|
tags: |
|
|
- Context |
|
|
- Qwen2.5-1.5B-Instruct-GPTQ-INT8 |
|
|
- Qwen2.5-1.5B-Instruct-GPTQ-INT4 |
|
|
--- |
|
|
|
|
|
# Qwen2.5-1.5B-Instruct |
|
|
|
|
|
This version of Qwen2.5-1.5B-Instruct has been converted to run on the Axera NPU using **w8a16** and **w4a16** quantization. |
|
|
|
|
|
This model has been optimized with the following LoRA: |
|
|
|
|
|
Compatible with Pulsar2 version: 4.1 |
|
|
|
|
|
## Feature |
|
|
|
|
|
- Support for longer contexts, in this sample it's 2.5k |
|
|
- Support context dialogue |
|
|
- System prompt kvcache is supported |
|
|
|
|
|
## Convert tools links: |
|
|
|
|
|
For those who are interested in model conversion, you can try to export axmodel through the original repo : https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GPTQ-Int8 |
|
|
|
|
|
[Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html) |
|
|
|
|
|
[AXera NPU AXEngine LLM Runtime](https://github.com/AXERA-TECH/ax-llm/tree/ax-context) |
|
|
|
|
|
[AXera NPU AXCL LLM Runtime](https://github.com/AXERA-TECH/ax-llm/tree/axcl-context) |
|
|
|
|
|
### Convert script |
|
|
|
|
|
The follow show how to convert Qwen2.5-1.5B-Instruct-GPTQ-Int8 |
|
|
|
|
|
``` |
|
|
pulsar2 llm_build --input_path Qwen/Qwen2.5-1.5B-Instruct-GPTQ-Int8 \ |
|
|
--output_path Qwen/Qwen2.5-1.5B-Instruct-GPTQ-Int8-ctx-ax650 \ |
|
|
--hidden_state_type bf16 --kv_cache_len 2047 --prefill_len 128 \ |
|
|
--last_kv_cache_len 128 \ |
|
|
--last_kv_cache_len 256 \ |
|
|
--last_kv_cache_len 384 \ |
|
|
--last_kv_cache_len 512 \ |
|
|
--last_kv_cache_len 640 \ |
|
|
--last_kv_cache_len 768 \ |
|
|
--last_kv_cache_len 896 \ |
|
|
--last_kv_cache_len 1024 \ |
|
|
--chip AX650 -c 1 --parallel 8 |
|
|
``` |
|
|
|
|
|
## Support Platform |
|
|
|
|
|
- AX650 |
|
|
- AX650N DEMO Board |
|
|
- [M4N-Dock(็ฑ่ฏๆดพPro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html) |
|
|
- [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html) |
|
|
- AX630C |
|
|
- *TBD* |
|
|
|
|
|
|Chips|w8a16|w4a16| DDR | Flash | |
|
|
|--|--|--|--|--| |
|
|
|AX650| 12 tokens/sec| 17 tokens/sec | 2.3GB | 2.3GB | |
|
|
|
|
|
## How to use |
|
|
|
|
|
Download all files from this repository to the device |
|
|
|
|
|
``` |
|
|
root@ax650:/mnt/qtang/llm-test/Qwen2.5-1.5B-Instruct# tree -L 1 |
|
|
. |
|
|
โโโ main_api |
|
|
โโโ main_ax650 |
|
|
โโโ main_axcl_aarch64 |
|
|
โโโ main_axcl_x86 |
|
|
โโโ post_config.json |
|
|
โโโ qwen2.5-1.5b-ctx-ax650 |
|
|
โโโ qwen2.5-1.5b-ctx-int4-ax650 |
|
|
โโโ qwen2.5_tokenizer |
|
|
โโโ qwen2.5_tokenizer_uid.py |
|
|
โโโ run_qwen2.5_1.5b_ctx_ax650_api.sh |
|
|
โโโ run_qwen2.5_1.5b_ctx_ax650.sh |
|
|
โโโ run_qwen2.5_1.5b_ctx_axcl_aarch64.sh |
|
|
โโโ run_qwen2.5_1.5b_ctx_axcl_x86.sh |
|
|
โโโ run_qwen2.5_1.5b_ctx_int4_ax650.sh |
|
|
``` |
|
|
|
|
|
#### Start the Tokenizer service |
|
|
|
|
|
``` |
|
|
root@ax650:/mnt/qtang/llm-test/Qwen2.5-1.5B-Instruct# python qwen2.5_tokenizer_uid.py |
|
|
Server running at http://0.0.0.0:12345 |
|
|
``` |
|
|
|
|
|
#### System prompt cache |
|
|
|
|
|
- The System prompt can be preset through the configuration file from `--system_prompt` |
|
|
- The System prompt can be cached in the form of kv cache to a specified folder for quick loading at the next run time from `--kvcache_path` |
|
|
- This folder needs to be created manually before running, for example `mkdir kvcache` |
|
|
|
|
|
``` |
|
|
root@ax650:/mnt/qtang/llm-test/qwen2.5-1.5b-ctx# cat run_qwen2.5_1.5b_ctx_ax650.sh |
|
|
./main_ax650 \ |
|
|
--template_filename_axmodel "qwen2.5-1.5b-ctx-ax650/qwen2_p128_l%d_together.axmodel" \ |
|
|
--axmodel_num 28 \ |
|
|
--tokenizer_type 2 \ |
|
|
--url_tokenizer_model "http://0.0.0.0:12345" \ |
|
|
--filename_post_axmodel "qwen2.5-1.5b-ctx-ax650/qwen2_post.axmodel" \ |
|
|
--filename_tokens_embed "qwen2.5-1.5b-ctx-ax650/model.embed_tokens.weight.bfloat16.bin" \ |
|
|
--tokens_embed_num 151936 \ |
|
|
--tokens_embed_size 1536 \ |
|
|
--use_mmap_load_embed 1 \ |
|
|
--live_print 1 |
|
|
|
|
|
#--system_prompt "ไฝ ็ๅๅญๅซๅฐๆบ๏ผallen๏ผ,ไฝ ๆฏไธไธชไบบ็ๆ ๅฎณ็AIๅฉๆใๆทฑๅณๅธไปๅคฉ๏ผ4ๆ1ๆฅ๏ผ้ดๅคฉ๏ผๆไบบ่๏ผๆฐๆธฉๅจ14ยฐC่ณ19ยฐCไน้ด๏ผๅพฎ้ฃใ" \ |
|
|
#--kvcache_path "./kvcache" \ |
|
|
``` |
|
|
|
|
|
#### Inference with AX650 Host, such as M4N-Dock(็ฑ่ฏๆดพPro) or AX650N DEMO Board |
|
|
|
|
|
Open another terminal and run `run_qwen2.5_1.5b_ctx_ax650.sh` |
|
|
|
|
|
``` |
|
|
root@ax650:/mnt/qtang/llm-test/qwen2.5-1.5b-ctx# ./run_qwen2.5_1.5b_ctx_ax650.sh |
|
|
[I][ Init][ 110]: LLM init start |
|
|
[I][ Init][ 34]: connect http://0.0.0.0:12345 ok |
|
|
[I][ Init][ 57]: uid: 1d0fadb4-1aa1-44d2-9587-e27badcd2ebf |
|
|
bos_id: -1, eos_id: 151645 |
|
|
3% | โโ | 1 / 31 [4.80s<148.95s, 0.21 count/s] tokenizer init ok |
|
|
[I][ Init][ 26]: LLaMaEmbedSelector use mmap |
|
|
100% | โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ | 31 / 31 [24.90s<24.90s, 1.24 count/s] init post axmodel ok,remain_cmm(7477 MB) |
|
|
[I][ Init][ 188]: max_token_len : 2047 |
|
|
[I][ Init][ 193]: kv_cache_size : 256, kv_cache_num: 2047 |
|
|
[I][ Init][ 201]: prefill_token_num : 128 |
|
|
[I][ Init][ 205]: grp: 1, prefill_max_token_num : 1 |
|
|
[I][ Init][ 205]: grp: 2, prefill_max_token_num : 128 |
|
|
[I][ Init][ 205]: grp: 3, prefill_max_token_num : 256 |
|
|
[I][ Init][ 205]: grp: 4, prefill_max_token_num : 384 |
|
|
[I][ Init][ 205]: grp: 5, prefill_max_token_num : 512 |
|
|
[I][ Init][ 205]: grp: 6, prefill_max_token_num : 640 |
|
|
[I][ Init][ 205]: grp: 7, prefill_max_token_num : 768 |
|
|
[I][ Init][ 205]: grp: 8, prefill_max_token_num : 896 |
|
|
[I][ Init][ 205]: grp: 9, prefill_max_token_num : 1024 |
|
|
[I][ Init][ 209]: prefill_max_token_num : 1024 |
|
|
[I][ load_config][ 282]: load config: |
|
|
{ |
|
|
"enable_repetition_penalty": false, |
|
|
"enable_temperature": false, |
|
|
"enable_top_k_sampling": false, |
|
|
"enable_top_p_sampling": false, |
|
|
"penalty_window": 20, |
|
|
"repetition_penalty": 1.2, |
|
|
"temperature": 0.9, |
|
|
"top_k": 10, |
|
|
"top_p": 0.8 |
|
|
} |
|
|
|
|
|
[I][ Init][ 218]: LLM init ok |
|
|
Type "q" to exit, Ctrl+c to stop current running |
|
|
[I][ GenerateKVCachePrefill][ 271]: input token num : 21, prefill_split_num : 1 prefill_grpid : 2 |
|
|
[I][ GenerateKVCachePrefill][ 308]: input_num_token:21 |
|
|
[I][ main][ 230]: precompute_len: 21 |
|
|
[I][ main][ 231]: system_prompt: |
|
|
prompt >> who are you? |
|
|
[I][ SetKVCache][ 531]: prefill_grpid:2 kv_cache_num:128 precompute_len:21 input_num_token:12 |
|
|
[I][ SetKVCache][ 534]: current prefill_max_token_num:896 |
|
|
[I][ Run][ 660]: input token num : 12, prefill_split_num : 1 |
|
|
[I][ Run][ 686]: input_num_token:12 |
|
|
[I][ Run][ 829]: ttft: 306.20 ms |
|
|
I am Qwen, a large language model created by Alibaba Cloud. I am here to assist you with your questions and provide helpful information. How may I assist you today? |
|
|
|
|
|
[N][ Run][ 943]: hit eos,avg 12.20 token/s |
|
|
|
|
|
[I][ GetKVCache][ 500]: precompute_len:68, remaining:956 |
|
|
prompt >> q |
|
|
root@ax650:/mnt/qtang/llm-test/qwen2.5-1.5b-ctx# |
|
|
``` |
|
|
|
