DeepSeek-R1-Distill-Qwen-1.5B — RKLLM build for RK3588 boards
Built with DeepSeek (DeepSeek License Agreement)
Author: @jamescallander
Source model: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B · Hugging Face
Target: Rockchip RK3588 NPU via RKNN-LLM Runtime
This repository hosts a conversion of
DeepSeek-R1-Distill-Qwen-1.5B
for use on Rockchip RK3588 single-board computers (Orange Pi 5 plus, Radxa Rock 5b+, Banana Pi M7, etc.). Conversion was performed using the RKNN-LLM toolkit
Conversion details
- RKLLM-Toolkit version: v1.2.1
- NPU driver: v0.9.8
- Python: 3.12
- Quantization:
w8a8_g128
- Output: single-file
.rkllm
artifact - Tokenizer: not required at runtime (UI handles prompt I/O)
Intended use
- On-device inference on RK3588 SBCs.
- Reasoning-focused model — designed to handle multi-step thinking, problem-solving, and structured explanations.
- Well-suited for tasks that need step-by-step reasoning or more careful breakdowns than typical instruction models.
Limitations
- Requires 2GB free memory
- Quantized build (
w8a8_g128
) may show small quality differences vs. full-precision upstream. - Tested on Radxa Rock 5B+; other devices may require different drivers/toolkit versions.
- While strong at reasoning, performance is limited by RK3588’s NPU compared to high-end GPUs.
Quick start (RK3588)
1) Install runtime
The RKNN-LLM toolkit and instructions can be found on the specific development board's manufacturer website or from airockchip's github page.
Download and install the required packages as per the toolkit's instructions.
2) Simple Flask server deployment
The simplest way the deploy the .rkllm
converted model is using an example script provided in the toolkit in this directory: rknn-llm/examples/rkllm_server_demo
python3 <TOOLKIT_PATH>/rknn-llm/examples/rkllm_server_demo/flask_server.py \
--rkllm_model_path <MODEL_PATH>/DeepSeek-R1-Distill-Qwen-1.5B_w8a8_g128_rk3588.rkllm \
--target_platform rk3588
3) Sending a request
A basic format for message request is:
{
"model":"DeepSeek-R1-Distill-Qwen-1.5B",
"messages":[{
"role":"user",
"content":"<YOUR_PROMPT_HERE>"}],
"stream":false
}
Example request using curl
:
curl -s -X POST <SERVER_IP_ADDRESS>:8080/rkllm_chat \
-H 'Content-Type: application/json' \
-d '{"model":"DeepSeek-R1-Distill-Qwen-1.5B","messages":[{"role":"user","content":"In 2 or 3 sentences, who was Napoleon Bonaparte?"}],"stream":false}'
The response is formated in the following way:
{
"choices":[{
"finish_reason":"stop",
"index":0,
"logprobs":null,
"message":{
"content":"<MODEL_REPLY_HERE">,
"role":"assistant"}}],
"created":null,
"id":"rkllm_chat",
"object":"rkllm_chat",
"usage":{
"completion_tokens":null,
"prompt_tokens":null,
"total_tokens":null}
}
Example response:
{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"content":"Okay, so I need to figure out who Napoleon Bonaparte is. I remember he's a French leader, but I'm not exactly sure about all the details. Let me think... He was born in 1795, right? And he died in 1881. That seems like a long time! I think he was a military leader and played a big role during the French Revolution. Wait, how did he get involved with the revolution? I believe he was a member of the National Assembly, which is where the French government is elected. Maybe he was one of the first people to support the revolution. He probably led some of the revolutionary activities, like the establishment of the People's Republic of China or something similar. I also remember that Napoleon was known for his strong sense of nationalism and his ability to unite a large population. His policies during the Revolution included establishing a republic with a weak central government, which he called the "Patriots' Republic." He probably wanted to eliminate corruption and strengthen the country's sovereignty. In terms of his political stance, I think he was a proponent of the French Nationalists party, which was a major political group in France during that time. His actions were aimed at maintaining French unity and preventing foreign interference in the revolution. So putting it all together, Napoleon Bonaparte was a key figure in the French Revolution, known for his military leadership, strong nationalism, and policies that aimed to unite France under a republic with a weak central government. </think> Napoleon Bonaparte was a prominent French leader who played a pivotal role during the French Revolution. Born in 1795 and dying in 1881, he was a member of the National Assembly and a key proponent of the French Nationalists party. Known for his military leadership and strong nationalism, Napoleon sought to unify France under a republic with a weak central government, opposing foreign interference and corruption. His policies aimed to maintain French unity and prevent revolutionaries from being suppressed.","role":"assistant"}}],"created":null,"id":"rkllm_chat","object":"rkllm_chat","usage":{"completion_tokens":null,"prompt_tokens":null,"total_tokens":null}}
Note on reasoning traces
This model outputs intermediate reasoning text (e.g., chains of thought) before its final response, enclosed by </think>
markers.
- Many OpenAI-compatible UIs automatically suppress or hide this internal reasoning.
- If your client does not, you may see the reasoning steps along with the final answer.
4) UI compatibility
This server exposes an OpenAI-compatible Chat Completions API.
You can connect it to any OpenAI-compatible client or UI (for example: Open WebUI)
- Configure your client with the API base:
http://<SERVER_IP_ADDRESS>:8080
and use the endpoint:/rkllm_chat
- Make sure the
model
field matches the converted model’s name, for example:
{
"model": "DeepSeek-R1-Distill-Qwen-1.5B",
"messages": [{"role":"user","content":"Hello!"}],
"stream": false
}
License
This conversion follows the MIT License
- Attribution: Built with DeepSeek-R1-Distill-Qwen-1.5B (DeepSeek-AI)
- Required notice: see
NOTICE
- Modifications: quantization (w8a8_g128), export to
.rkllm
format for RK3588 SBCs
- Downloads last month
- 10
Model tree for jamescallander/DeepSeek-R1-Distill-Qwen-1.5B_w8a8_g128_rk3588.rkllm
Base model
deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B