Instructions to use webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M

Use Docker

docker model run hf.co/webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M

Ollama
How to use webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF with Ollama:
```
ollama run hf.co/webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M
```
Unsloth Desktop

How to use webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF with Docker Model Runner:
```
docker model run hf.co/webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M
```

Lemonade

How to use webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Sakura-Sage-A1.5-Q4_K_M-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Sakura Sage-A1.5 Q4_K_M GGUF

Q4_K_M quantization of InternScience/Sage-A1.5, built directly from the original BF16 safetensors, including its built-in MTP head. The language weights were converted with llama.cpp and quantized on CPU.

Source and format

Item	Details
Original model	InternScience/Sage-A1.5
Source revision	`0e3205cf441cf01df0c0d238cc4738d0e43b4ca7`
Source precision	BF16, with original F32 tensors where provided
Output	Q4_K_M, standard mixed tensor quantization
Architecture	Qwen3.5 MoE / `qwen35moe`
Blocks	40 language blocks + 1 MTP block
Experts	256 per MoE layer, 8 selected per token
Tokenizer and chat template	From the pinned original checkpoint
Conversion and quantizer	llama.cpp `970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc`
License	Apache 2.0

This is a full language-model quantization. The official Q8_0 GGUF was not used as the input to the language-weight quantizer. Norms, routing and other tensors follow the upstream converter and quantizer's precision rules; Q4_K_M does not mean every tensor uses the same bit width.

Size comparison

Distribution	Scope	Size
Original BF16 checkpoint	Complete upstream safetensors, including vision and MTP	71.908 GB / 66.970 GiB
Official Sage-A1.5 Q8_0 GGUF	Language model + MTP	37.802 GB / 35.206 GiB
Sakura Q4_K_M	Language model + MTP	21.713 GB / 20.222 GiB
Sage-A1.5 mmproj	Separate, unchanged official projector	0.899 GB / 0.838 GiB

The main Q4_K_M file is 42.56% smaller than the official Q8_0 main file, saving 16.089 GB. With the optional projector, this package totals 22.613 GB. File sizes are not a promise of runtime memory usage; context and backend buffers require additional memory.

Files

File	Purpose
`Sakura-Sage-A1.5-Q4_K_M.gguf`	Quantized language model, with MTP weights
`Sage-A1.5-mmproj.gguf`	Original official vision projector, copied unchanged
`SHA256SUMS`	SHA256 checksums for both GGUF files
`build-manifest.json`	Pinned provenance, tensor types, sizes and checksums

Main file SHA256: 4399beb2854d11599c5a5d45b92dd956b1855fe2452b455e4a922f42b7e59ff1.

Usage

Use a recent llama.cpp build supporting Qwen3.5 MoE and its included MTP tensors. The conversion revision above is the exact format reference. Backend acceleration and MTP usage depend on the runtime and its settings.

hf download webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF Sakura-Sage-A1.5-Q4_K_M.gguf --local-dir .
llama-server -m Sakura-Sage-A1.5-Q4_K_M.gguf -c 32768

For image input, also download Sage-A1.5-mmproj.gguf from this repository and pass --mmproj Sage-A1.5-mmproj.gguf to a compatible multimodal runtime. Text-only use does not require the projector. The original model supports a longer context; choose a context size that fits your available memory.

The original model's recommended prompt format and capabilities are documented in the upstream model card. The official Q8_0 distribution supplies the unchanged projector and the Q8_0 size reference.

Local performance reference

The model owner reported the following results for a separate local artifact, Sage-A1.5-Q5_K_M-imatrix.gguf (25,628,142,112 bytes / 23.868 GiB). It was requantized from the official Q8_0 file with an importance matrix; its 20 block-40 MTP tensors were reported as byte-identical Q8_0/F32 copies. These figures apply to that reference artifact. They do not establish performance for the BF16-generated files in this repository.

Build detail	This BF16 cloud release	Owner's local reference
Language-weight input	Original BF16 safetensors	Official Q8_0 GGUF
Importance matrix	None; standard llama.cpp quantization	96 x 512-token mixed calibration
MTP precision	Standard mixed quantization policy, plus retained F32 tensors	Original Q8_0/F32, 20 tensors reported unchanged
Reference file size	See this release's size table	25.628 GB

Reported configuration: Ryzen AI Max+ 392 / Radeon 8060S, 64 GB unified memory, Windows, Vulkan, llama.cpp build 11594 (e0bcaccbe, Clang 20.1.8), full GPU offload, Flash Attention, a 65,536-token context allocation, one sequence, and draft-mtp with at most two draft tokens.

Metric / workload	Owner-reported result
Prompt processing	850-1,050 tokens/s
TTFT, short prompts	0.6-0.9 s
TTFT, 8,500-token input	8-12 s
Reasoning decode	59-65 tokens/s
Reasoning MTP acceptance	58-64%
Python generation decode	64-77 tokens/s
Python MTP acceptance	Up to 87.5%
Python decode gain over MTP off	13-34%
Model and KV-cache UMA usage	Approximately 27.8 GB

The owner reported successful loading, web UI, OpenAI-compatible chat API, streaming and tool/JSON calls for text workloads. The reported calibration covered all 40 main blocks and all 256 experts per MoE layer, with zero unseen experts.

Local reference provenance and exact commands

Q8_0 source: InternScience/Sage-A1.5-Q8_0-GGUF, revision dfcef699cb670918f3aa4fa6462559b16294fc8f.

Source SHA256: f7247f6faef3f6c616b953595e1fc7c3fbf257d02d9915af5fdf15e808ea8ee4.

Local reference SHA256: c4ff7f8f46dbb1cc4b9a9a85be111033ed658880aaddf1cb0d346c5de58cae63.

Importance matrix: Sage-A1.5-Q8_0-calib-mix-96x512-main.imatrix.gguf, 384,359,808 bytes, from calib_mix_v1.txt (96 chunks of 512 tokens). Commands below reproduce the reported local configuration and refer to that runtime build.

llama-quantize.exe --allow-requantize --imatrix Sage-A1.5-Q8_0-calib-mix-96x512-main.imatrix.gguf --tensor-type "^blk[.]40[.].*=q8_0" --max-buffer-size 2048 Sage-A1.5-Q8_0.gguf Sage-A1.5-Q5_K_M-imatrix.gguf Q5_K_M 12

llama-server.exe -m Sage-A1.5-Q5_K_M-imatrix.gguf --alias sage-a15-q5-test --host 127.0.0.1 --port 18105 -ngl 99 --device Vulkan0 -c 65536 -b 512 -ub 512 --parallel 1 -fa on --jinja --reasoning-format deepseek --spec-type draft-mtp --spec-draft-n-max 2 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0

中文说明

Sakura Sage-A1.5 Q4_K_M GGUF

本仓库基于 InternScience/Sage-A1.5 的原始 BF16 safetensors 直接转换并进行 CPU 量化，包含模型自带的 MTP 预测头。语言模型权重并非从 Q8_0 文件再次量化。

源版本为 0e3205cf441cf01df0c0d238cc4738d0e43b4ca7，llama.cpp 版本为 970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc。架构为 qwen35moe，包含 40 个语言模型块及 1 个 MTP 块。Q4_K_M 使用上游标准混合精度规则，归一化、路由等张量保留上游规定的精度。

文件	大小
官方 Q8_0 主模型（含 MTP）	37.802 GB
本仓库 Q4_K_M 主模型（含 MTP）	21.713 GB
原样保留的官方视觉投影器	0.899 GB

主模型比官方 Q8_0 主文件减少 **42.56%**，节省 16.089 GB。这些数字表示文件大小；实际运行还需要上下文及后端缓冲区内存。

使用支持 Qwen3.5 MoE 的近期 llama.cpp 版本。纯文本推理只需主 GGUF 文件；图像输入还需要 Sage-A1.5-mmproj.gguf。MTP 的启用方式取决于所用运行时。分词器和聊天模板来自固定版本的原始模型。许可证为 Apache 2.0。

本地性能参考

模型所有者报告了独立本地文件 Sage-A1.5-Q5_K_M-imatrix.gguf 的文本运行数据：该文件由官方 Q8_0 通过重要性矩阵重新量化，大小 25.628 GB，20 个 MTP 张量据报告保持原始 Q8_0/F32 字节不变。本仓库从原始 BF16 生成，未使用该重要性矩阵，MTP 按标准混合精度规则处理。下列数据仅对应该本地参考文件。

硬件为 Ryzen AI Max+ 392、Radeon 8060S、64 GB 统一内存，Windows/Vulkan，llama.cpp build 11594（e0bcaccbe），全 GPU 卸载、Flash Attention、65,536 token 上下文分配、单序列、draft-mtp 最多 2 个草稿 token。

指标	所有者报告
提示处理	850-1,050 tokens/s
短提示首 token 延迟	0.6-0.9 秒
8,500-token 输入首 token 延迟	8-12 秒
推理解码	59-65 tokens/s，MTP 接受率 58-64%
Python 代码生成	64-77 tokens/s，MTP 接受率最高 87.5%
Python 解码相对关闭 MTP 的提升	13-34%
模型与 KV 缓存内存占用	约 27.8 GB UMA

所有者报告文本加载、Web UI、聊天 API、流式输出和工具/JSON 调用正常。校准据报告覆盖全部 40 个主模型块、每个 MoE 层全部 256 个专家，未见专家数为 0。

Downloads last month: 418

GGUF

Model size

36B params

Architecture

qwen35moe

Hardware compatibility

4-bit

Model tree for webmp3/Sakura-Sage-A1.5-Q4_K_M-GGUF

Base model

InternScience/Sage-A1.5

Quantized

(45)

this model