--- library_name: transformers license: bsd-3-clause base_model: - Qwen/Qwen2.5-0.5B-Instruct-GPTQ-Int8 tags: - Qwen - Qwen2.5-0.5B-Instruct - Qwen2.5-0.5B-Instruct-GPTQ-Int8 - GPTQ language: - en --- # Qwen2.5-0.5B-Instruct-GPTQ-Int8 This version of Qwen2.5-0.5B-Instruct-GPTQ-Int8 has been converted to run on the Axera NPU using **w8a16** quantization. This model has been optimized with the following LoRA: Compatible with Pulsar2 version: 4.0-patch1 ## Convert tools links: For those who are interested in model conversion, you can try to export axmodel through the original repo : https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GPTQ-Int8 [Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html) [AXera NPU LLM Runtime](https://github.com/AXERA-TECH/ax-llm) ## Support Platform - AX650 - AX650N DEMO Board - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html) - [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html) - AX630C - *developing* |Chips|w8a16|w4a16| |--|--|--| |AX650| 30 tokens/sec| TBD | ## How to use Download all files from this repository to the device ``` root@ax650:/mnt/qtang/llm-test/qwen2.5-0.5b-ctx# tree -L 1 . |-- main_ax650 |-- main_axcl_aarch64 |-- main_axcl_x86 |-- post_config.json |-- qwen2.5-0.5b-gptq-int8-ctx-ax630c |-- qwen2.5-0.5b-gptq-int8-ctx-ax650 |-- qwen2.5_tokenizer |-- qwen2.5_tokenizer_uid.py |-- run_qwen2.5_0.5b_gptq_int8_ctx_ax630c.sh `-- run_qwen2.5_0.5b_gptq_int8_ctx_ax650.sh 3 directories, 7 files ``` #### Start the Tokenizer service ``` root@ax650:/mnt/qtang/llm-test/qwen2.5-0.5b-ctx# python3 qwen2.5_tokenizer_uid.py Server running at http://0.0.0.0:12345 ``` #### Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board Open another terminal and run `run_qwen2.5_0.5b_gptq_int8_ax650.sh` ``` root@ax650:/mnt/qtang/llm-test/qwen2.5-0.5b-ctx# ./run_qwen2.5_0.5b_gptq_int8_ctx_ax650.sh [I][ Init][ 110]: LLM init start [I][ Init][ 34]: connect http://127.0.0.1:12345 ok [I][ Init][ 57]: uid: cdeaf62e-0243-4dc9-b557-23a7c1ba7da1 bos_id: -1, eos_id: 151645 100% | ████████████████████████████████ | 27 / 27 [12.35s<12.35s, 2.19 count/s] init post axmodel ok,remain_cmm(3960 MB) [I][ Init][ 188]: max_token_len : 2560 [I][ Init][ 193]: kv_cache_size : 128, kv_cache_num: 2560 [I][ Init][ 201]: prefill_token_num : 128 [I][ Init][ 205]: grp: 1, prefill_max_token_num : 1 [I][ Init][ 205]: grp: 2, prefill_max_token_num : 128 [I][ Init][ 205]: grp: 3, prefill_max_token_num : 512 [I][ Init][ 205]: grp: 4, prefill_max_token_num : 1024 [I][ Init][ 205]: grp: 5, prefill_max_token_num : 1536 [I][ Init][ 205]: grp: 6, prefill_max_token_num : 2048 [I][ Init][ 209]: prefill_max_token_num : 2048 [I][ load_config][ 282]: load config: { "enable_repetition_penalty": false, "enable_temperature": false, "enable_top_k_sampling": true, "enable_top_p_sampling": false, "penalty_window": 20, "repetition_penalty": 1.2, "temperature": 0.9, "top_k": 1, "top_p": 0.8 } [I][ Init][ 218]: LLM init ok Type "q" to exit, Ctrl+c to stop current running [I][ GenerateKVCachePrefill][ 271]: input token num : 21, prefill_split_num : 1 prefill_grpid : 2 [I][ GenerateKVCachePrefill][ 308]: input_num_token:21 [I][ main][ 230]: precompute_len: 21 [I][ main][ 231]: system_prompt: You are Qwen, created by Alibaba Cloud. You are a helpful assistant. prompt >> who are you? [I][ SetKVCache][ 531]: prefill_grpid:2 kv_cache_num:128 precompute_len:38 input_num_token:12 [I][ SetKVCache][ 534]: current prefill_max_token_num:1920 [I][ Run][ 660]: input token num : 12, prefill_split_num : 1 [I][ Run][ 686]: input_num_token:12 [I][ Run][ 829]: ttft: 134.80 ms I am Qwen, a large language model created by Alibaba Cloud. I am designed to assist with a wide range of tasks, from general knowledge to specific areas such as science, technology, and more. How can I help you today? [N][ Run][ 943]: hit eos,avg 30.88 token/s [I][ GetKVCache][ 500]: precompute_len:98, remaining:1950 prompt >> what can you do? [I][ SetKVCache][ 531]: prefill_grpid:2 kv_cache_num:128 precompute_len:98 input_num_token:13 [I][ SetKVCache][ 534]: current prefill_max_token_num:1920 [I][ Run][ 660]: input token num : 13, prefill_split_num : 1 [I][ Run][ 686]: input_num_token:13 [I][ Run][ 829]: ttft: 134.97 ms I can answer questions, provide information, assist with tasks, and even engage in creative writing. I'm here to help you with any questions or tasks you might have! [N][ Run][ 943]: hit eos,avg 30.85 token/s [I][ GetKVCache][ 500]: precompute_len:145, remaining:1903 ```