AXERA-TECH/Qwen2.5-0.5B-Instruct-CTX-Int8

Qwen2.5-0.5B-Instruct-GPTQ-Int8

This version of Qwen2.5-0.5B-Instruct-GPTQ-Int8 has been converted to run on the Axera NPU using w8a16 quantization.

This model has been optimized with the following LoRA:

Compatible with Pulsar2 version: 4.0-patch1

Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo : https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GPTQ-Int8

Pulsar2 Link, How to Convert LLM from Huggingface to axmodel

AXera NPU LLM Runtime

Support Platform

• AX650
  • AX650N DEMO Board
  • M4N-Dock(爱芯派Pro)
  • M.2 Accelerator card
• AX630C
  • developing

Chips	w8a16	w4a16
AX650	30 tokens/sec	TBD

How to use

Download all files from this repository to the device

root@ax650:/mnt/qtang/llm-test/qwen2.5-0.5b-ctx# tree -L 1
.
|-- main_ax650
|-- main_axcl_aarch64
|-- main_axcl_x86
|-- post_config.json
|-- qwen2.5-0.5b-gptq-int8-ctx-ax630c
|-- qwen2.5-0.5b-gptq-int8-ctx-ax650
|-- qwen2.5_tokenizer
|-- qwen2.5_tokenizer_uid.py
|-- run_qwen2.5_0.5b_gptq_int8_ctx_ax630c.sh
`-- run_qwen2.5_0.5b_gptq_int8_ctx_ax650.sh

3 directories, 7 files

Start the Tokenizer service

root@ax650:/mnt/qtang/llm-test/qwen2.5-0.5b-ctx# python3 qwen2.5_tokenizer_uid.py
Server running at http://0.0.0.0:12345

Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board

Open another terminal and run run_qwen2.5_0.5b_gptq_int8_ax650.sh

root@ax650:/mnt/qtang/llm-test/qwen2.5-0.5b-ctx# ./run_qwen2.5_0.5b_gptq_int8_ctx_ax650.sh
[I][                            Init][ 110]: LLM init start
[I][                            Init][  34]: connect http://127.0.0.1:12345 ok
[I][                            Init][  57]: uid: cdeaf62e-0243-4dc9-b557-23a7c1ba7da1
bos_id: -1, eos_id: 151645
100% | ████████████████████████████████ |  27 /  27 [12.35s<12.35s, 2.19 count/s] init post axmodel ok,remain_cmm(3960 MB)
[I][                            Init][ 188]: max_token_len : 2560
[I][                            Init][ 193]: kv_cache_size : 128, kv_cache_num: 2560
[I][                            Init][ 201]: prefill_token_num : 128
[I][                            Init][ 205]: grp: 1, prefill_max_token_num : 1
[I][                            Init][ 205]: grp: 2, prefill_max_token_num : 128
[I][                            Init][ 205]: grp: 3, prefill_max_token_num : 512
[I][                            Init][ 205]: grp: 4, prefill_max_token_num : 1024
[I][                            Init][ 205]: grp: 5, prefill_max_token_num : 1536
[I][                            Init][ 205]: grp: 6, prefill_max_token_num : 2048
[I][                            Init][ 209]: prefill_max_token_num : 2048
[I][                     load_config][ 282]: load config:
{
    "enable_repetition_penalty": false,
    "enable_temperature": false,
    "enable_top_k_sampling": true,
    "enable_top_p_sampling": false,
    "penalty_window": 20,
    "repetition_penalty": 1.2,
    "temperature": 0.9,
    "top_k": 1,
    "top_p": 0.8
}

[I][                            Init][ 218]: LLM init ok
Type "q" to exit, Ctrl+c to stop current running
[I][          GenerateKVCachePrefill][ 271]: input token num : 21, prefill_split_num : 1 prefill_grpid : 2
[I][          GenerateKVCachePrefill][ 308]: input_num_token:21
[I][                            main][ 230]: precompute_len: 21
[I][                            main][ 231]: system_prompt: You are Qwen, created by Alibaba Cloud. You are a helpful assistant.
prompt >> who are you?
[I][                      SetKVCache][ 531]: prefill_grpid:2 kv_cache_num:128 precompute_len:38 input_num_token:12
[I][                      SetKVCache][ 534]: current prefill_max_token_num:1920
[I][                             Run][ 660]: input token num : 12, prefill_split_num : 1
[I][                             Run][ 686]: input_num_token:12
[I][                             Run][ 829]: ttft: 134.80 ms
I am Qwen, a large language model created by Alibaba Cloud. I am designed to assist with a wide range of tasks,
from general knowledge to specific areas such as science, technology, and more. How can I help you today?

[N][                             Run][ 943]: hit eos,avg 30.88 token/s

[I][                      GetKVCache][ 500]: precompute_len:98, remaining:1950
prompt >> what can you do?
[I][                      SetKVCache][ 531]: prefill_grpid:2 kv_cache_num:128 precompute_len:98 input_num_token:13
[I][                      SetKVCache][ 534]: current prefill_max_token_num:1920
[I][                             Run][ 660]: input token num : 13, prefill_split_num : 1
[I][                             Run][ 686]: input_num_token:13
[I][                             Run][ 829]: ttft: 134.97 ms
I can answer questions, provide information, assist with tasks, and even engage in creative writing.
I'm here to help you with any questions or tasks you might have!

[N][                             Run][ 943]: hit eos,avg 30.85 token/s

[I][                      GetKVCache][ 500]: precompute_len:145, remaining:1903