|
|
--- |
|
|
license: apache-2.0 |
|
|
base_model: Qwen/Qwen2.5-1.5B-Instruct |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- chat |
|
|
--- |
|
|
|
|
|
# litert-community/Qwen2.5-1.5B-Instruct |
|
|
|
|
|
This model provides a few variants of |
|
|
[Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) that are ready for |
|
|
deployment on Android using the |
|
|
[LiteRT (fka TFLite) stack](https://ai.google.dev/edge/litert) and |
|
|
[MediaPipe LLM Inference API](https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference). |
|
|
|
|
|
## Use the models |
|
|
|
|
|
### Colab |
|
|
|
|
|
*Disclaimer: The target deployment surface for the LiteRT models is |
|
|
Android/iOS/Web and the stack has been optimized for performance on these |
|
|
targets. Trying out the system in Colab is an easier way to familiarize yourself |
|
|
with the LiteRT stack, with the caveat that the performance (memory and latency) |
|
|
on Colab could be much worse than on a local device.* |
|
|
|
|
|
[](https://colab.research.google.com/#fileId=https://huggingface.co/litert-community/Qwen2.5-1.5B-Instruct/blob/main/notebook.ipynb) |
|
|
|
|
|
### Android |
|
|
|
|
|
* Download and install |
|
|
[the apk](https://github.com/google-ai-edge/gallery/releases/latest/download/ai-edge-gallery.apk). |
|
|
* Follow the instructions in the app. |
|
|
|
|
|
To build the demo app from source, please follow the [instructions](https://github.com/google-ai-edge/gallery/blob/main/README.md) |
|
|
from the GitHub repository. |
|
|
|
|
|
### iOS |
|
|
|
|
|
* Clone the [MediaPipe samples](https://github.com/google-ai-edge/mediapipe-samples) |
|
|
repository and follow the [instructions](https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/llm_inference/ios/README.md) |
|
|
to build the LLM Inference iOS Sample App using XCode. |
|
|
* Run the app via the iOS simulator or deploy to an iOS device. |
|
|
|
|
|
## Performance |
|
|
|
|
|
### Android |
|
|
|
|
|
Note that all benchmark stats are from a Samsung S24 Ultra and multiple prefill signatures enabled. |
|
|
|
|
|
<table border="1"> |
|
|
<tr> |
|
|
<th style="text-align: left">Backend</th> |
|
|
<th style="text-align: left">Quantization scheme</th> |
|
|
<th style="text-align: left">Context length</th> |
|
|
<th style="text-align: left">Prefill (tokens/sec)</th> |
|
|
<th style="text-align: left">Decode (tokens/sec)</th> |
|
|
<th style="text-align: left">Time-to-first-token (sec)</th> |
|
|
<th style="text-align: left">CPU Memory (RSS in MB)</th> |
|
|
<th style="text-align: left">GPU Memory (RSS in MB)</th> |
|
|
<th style="text-align: left">Model size (MB)</th> |
|
|
<th></th> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="5"><p style="text-align: left">CPU</p></td> |
|
|
<td rowspan="3"><p style="text-align: left">fp32 (baseline)</p></td> |
|
|
<td><p style="text-align: right">1280</p></td> |
|
|
<td><p style="text-align: right">27 tk/s</p></td> |
|
|
<td><p style="text-align: right">6 tk/s</p></td> |
|
|
<td><p style="text-align: right">9.88 s</p></td> |
|
|
<td><p style="text-align: right">6,144 MB</p></td> |
|
|
<td><p style="text-align: right"></p></td> |
|
|
<td><p style="text-align: right">5,895 MB</p></td> |
|
|
<td><p style="text-align: left"><a style="text-decoration: none" href="https://huggingface.co/litert-community/Qwen2.5-1.5B-Instruct/resolve/main/Qwen2.5-1.5B-Instruct_multi-prefill-seq_f32_ekv1280.task">🔗</a></p></td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="2"><p style="text-align: right">1280</p></td> |
|
|
<td><p style="text-align: right">106 tk/s</p></td> |
|
|
<td><p style="text-align: right">23 tk/s</p></td> |
|
|
<td><p style="text-align: right">2.74 s</p></td> |
|
|
<td><p style="text-align: right">1,820 MB</p></td> |
|
|
<td><p style="text-align: right"></p></td> |
|
|
<td><p style="text-align: right">1,523 MB</p></td> |
|
|
<td><p style="text-align: left"><a style="text-decoration: none" href="https://huggingface.co/litert-community/Qwen2.5-1.5B-Instruct/resolve/main/Qwen2.5-1.5B-Instruct_multi-prefill-seq_q8_ekv1280.task">🔗</a></p></td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td><p style="text-align: right">63 tk/s</p></td> |
|
|
<td><p style="text-align: right">20 tk/s</p></td> |
|
|
<td><p style="text-align: right">4.40 s</p></td> |
|
|
<td><p style="text-align: right">2,042 MB</p></td> |
|
|
<td><p style="text-align: right"></p></td> |
|
|
<td><p style="text-align: right">1,523 MB</p></td> |
|
|
<td><p style="text-align: left"><a style="text-decoration: none" href="https://huggingface.co/litert-community/Qwen2.5-1.5B-Instruct/resolve/main/Qwen2.5-1.5B-Instruct_multi-prefill-seq_q8_ekv4096.task">🔗</a></p></td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td rowspan="2"><p style="text-align: left">dynamic_int8</p></td> |
|
|
<td rowspan="2"><p style="text-align: right">1280</p></td> |
|
|
<td><p style="text-align: right">706 tk/s</p></td> |
|
|
<td><p style="text-align: right">24 tk/s</p></td> |
|
|
<td><p style="text-align: right">6.94 s</p></td> |
|
|
<td><p style="text-align: right">3,175 MB</p></td> |
|
|
<td><p style="text-align: right">1,504 MB</p></td> |
|
|
<td><p style="text-align: right">1,523 MB</p></td> |
|
|
<td><p style="text-align: left"><a style="text-decoration: none" href="https://huggingface.co/litert-community/Qwen2.5-1.5B-Instruct/resolve/main/Qwen2.5-1.5B-Instruct_multi-prefill-seq_q8_ekv1280.task">🔗</a></p></td> |
|
|
</tr> |
|
|
<tr> |
|
|
<td><p style="text-align: right">417 tk/s</p></td> |
|
|
<td><p style="text-align: right">22 tk/s</p></td> |
|
|
<td><p style="text-align: right">7.93 s</p></td> |
|
|
<td><p style="text-align: right">3,176 MB</p></td> |
|
|
<td><p style="text-align: right">1,875 MB</p></td> |
|
|
<td><p style="text-align: right">1,523 MB</p></td> |
|
|
<td><p style="text-align: left"><a style="text-decoration: none" href="https://huggingface.co/litert-community/Qwen2.5-1.5B-Instruct/resolve/main/Qwen2.5-1.5B-Instruct_multi-prefill-seq_q8_ekv4096.task">🔗</a></p></td> |
|
|
</tr> |
|
|
|
|
|
</table> |
|
|
|
|
|
* For the list of supported quantization schemes see [supported-schemes](https://github.com/google-ai-edge/ai-edge-torch/tree/main/ai_edge_torch/generative/quantize#supported-schemes). |
|
|
For these models, we are using prefill signature lengths of 32, 128, 512 and 1280. |
|
|
* Model Size: measured by the size of the .tflite flatbuffer (serialization |
|
|
format for LiteRT models) |
|
|
* Memory: indicator of peak RAM usage |
|
|
* The inference on CPU is accelerated via the LiteRT |
|
|
[XNNPACK](https://github.com/google/XNNPACK) delegate with 4 threads |
|
|
* Benchmark is run with cache enabled and initialized. During the first run, |
|
|
the time to first token may differ. |
|
|
|
|
|
|
