---
license: mit
language:
- en
tags:
- Transformer
- Tracking
- ONNX
---

# MixFormerV2

This version of MixFormerV2 has been converted to run on the Axera NPU using **w8a16** quantization.

This model has been optimized with the following LoRA: 

Compatible with Pulsar2 version: 3.4

## Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through 

- [The repo of original](https://github.com/MCG-NJU/MixFormerV2)

- [The repo of AXera Platform](https://github.com/Jordan-5i/ax650_mixformer2_demo), which you can get the detial of guide

- [Pulsar2 Link, How to Convert ONNX to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/pulsar2/introduction.html) 


## Support Platform

- AX650
  - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)
  - [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html)
- AX630C
  - [爱芯派2](https://axera-pi-2-docs-cn.readthedocs.io/zh-cn/latest/index.html)
  - [Module-LLM](https://docs.m5stack.com/zh_CN/module/Module-LLM)
  - [LLM630 Compute Kit](https://docs.m5stack.com/zh_CN/core/LLM630%20Compute%20Kit)
 
|Chips|npu1|
|--|--|
|AX650| 11 ms |
|AX630C| 33 ms |

## How to use

Download all files from this repository to the device

```
root@ax650:/mnt/qtang/MixFormerV2# tree -L 1
.
├── ax650
├── car.avi
├── config.json
├── onnx
├── README.md
├── run_mixformer2_axmodel.py
└── run_mixformer2_onnx.py
```

### python env requirement

#### pyaxengine

https://github.com/AXERA-TECH/pyaxengine

```
wget https://github.com/AXERA-TECH/pyaxengine/releases/download/0.1.1rc0/axengine-0.1.1-py3-none-any.whl
pip install axengine-0.1.1-py3-none-any.whl
```

#### others

```
pip install argparse numpy opencv-python glob2
```

#### Inference with AX650 Host, such as M4N-Dock(爱芯派Pro)

```
root@ax650:/mnt/qtang/ax650_mixformer2_demo# python3 run_mixformer2_axmodel.py --model-path ax650/mixformer_v2.axmodel --frame-path car.avi -r 10
[INFO] Available providers:  ['AxEngineExecutionProvider']
[INFO] Using provider: AxEngineExecutionProvider
[INFO] Chip type: ChipType.MC50
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Engine version: 2.7.2a
[INFO] Model type: 0 (single core)
[INFO] Compiler version: 3.4-dirty 4ff37520-dirty
====================type================= [1079, 482] <class 'list'> <class 'list'>
第一帧初始化完毕！
Video: tracking     246.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Reached the maximum number of frames (10). Exiting loop.
video: average finale average tracking fps 31.8 fps
root@ax650:/mnt/qtang/ax650_mixformer2_demo#
```

#### Inference with M.2 Accelerator card

[What is M.2 Accelerator card?](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html), Show this DEMO based on Raspberry PI 5.

```
(axcl) axera@raspberrypi:~/samples/MixFormerV2 $ python3 run_mixformer2_axmodel.py --model-path ax650/mixformer_v2.axmodel --frame-path car.avi -r 10
[INFO] Available providers:  ['AXCLRTExecutionProvider']
[INFO] Using provider: AXCLRTExecutionProvider
[INFO] SOC Name: AX650N
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Compiler version: 3.4-dirty 4ff37520-dirty
====================type================= [1079, 482] <class 'list'> <class 'list'>
第一帧初始化完毕！
Video: tracking     925.0fps
Video: tracking     12.0fps
Video: tracking     12.0fps
Video: tracking     11.0fps
Video: tracking     11.0fps
Video: tracking     11.0fps
Video: tracking     11.0fps
Video: tracking     11.0fps
Video: tracking     10.0fps
Video: tracking     10.0fps
Video: tracking     10.0fps
Reached the maximum number of frames (10). Exiting loop.
video: average finale average tracking fps 114.9 fps
(axcl) axera@raspberrypi:~/samples/MixFormerV2 $
```