Instructions to use oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF # Run inference directly in the terminal: llama cli -hf oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF # Run inference directly in the terminal: llama cli -hf oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF # Run inference directly in the terminal: ./build/bin/llama cli -hf oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF
Use Docker
docker model run hf.co/oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF
- Llama new
- LM Studio
- Jan
- vLLM
How to use oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF
- Ollama
How to use oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF with Ollama:
ollama run hf.co/oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF
- Unsloth Desktop
- Pi
How to use oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF with Docker Model Runner:
docker model run hf.co/oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF
- Lemonade
How to use oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF
Run and chat with the model
lemonade run user.Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Qwen3.8-Flash-Next ยท GSQ-RCO ยท MoREL PB3-304-q2 ยท GGUF
Requires our llama.cpp fork, branch cnm/varexp, commit 7ed1a0a57e5621073146bca8d5c166e52cfe462b. Mainline llama.cpp will not load this model.
MoREL = Mixture of Ragged Experts per Layer: a physically pruned MoE with a different expert count in each layer. This 2-bit build is cut from the 512-expert IQ2_XS variant of ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF, using the PB3-304 mask. The mask keeps a mean of 304 experts per layer (range 234โ364, 48 routed layers), with top-10 routing unchanged. The quantization label describes the source release; individual tensor types are mixed. This is an independent community derivative, without endorsement from Qwen or ISTA-DASLab.
Downloads / prebuilt binaries
MoREL runtime cnm-varexp-v0.1 prerelease provides the fork runtime required by the PB3 MoREL models. It is also offered here for K192p-implant16 v2, whose generation with these packages has not been tested. These are machine-specific Ubuntu 24.04 packages. Choose the package that matches your CPU and backend; they are not universal Linux builds. Windows x64 CUDA/CPU, macOS arm64 and generic Linux x64 CUDA/Vulkan/CPU Actions packages are not available in this prerelease yet. Build from the release source for other machines.
| Machine / backend | Asset | SHA-256 |
|---|---|---|
| AVX2/FMA/F16C CPU (i10, Ryzen 5 5600G) | llama-cnm-varexp-v0.1-i10-cpu-linux-x86_64.tar.gz | 8c40381780f37427c52d7a5b30efe07bf0e0448ad0880f1a7d025ebf2ad0876e |
| AVX CPU + Radeon Vulkan (i3, i5-3570K / RX 580) | llama-cnm-varexp-v0.1-i3-vulkan-linux-x86_64.tar.gz | 5420ec8769075c1affe3b011b5a0d7dc92fa3cd688accfb7b8ee4b6e5b07f345 |
| CUDA 12.0 sm_75 (i5, RTX 2070) | llama-cnm-varexp-v0.1-i5-cuda-linux-x86_64.tar.gz | 7c7390f8193197fb38b8e3aa9ddd5507afa6caa21aaed19672e4c9857a8c502c |
| CUDA 12.0 sm_89/86/75 (i7, RTX 4060 Ti) | llama-cnm-varexp-v0.1-i7-cuda-linux-x86_64.tar.gz | 567d67ff6ca26c3046c39247b8efa704f897d981f2fa2671a6edd7ebe8a7e16b |
Each tarball has a matching .sha256 asset on the release page. Download both and run sha256sum -c <archive>.sha256 before extraction. Keep the executables and project shared libraries together. System C/C++ libraries and GPU drivers/CUDA libraries are external dependencies. These packages disable HTTPS and the embedded server Web UI.
The release notes contain the complete test matrix. Packaging checks covered all four builds; PB3-208 had short CPU (i10) and partial-offload Vulkan (i3) completions. CUDA generation, PB3-304/K192p generation with these packaged binaries, full Vulkan residency, long contexts and MTP were not tested in this release. Existing model-card evaluations used their separately recorded runtime configurations.
How it was made
- Record expert-routing tomography while an agent works through ABAP coding sessions, including trajectories produced by other models. Separate explore, edit, and verify phases.
- PB3 gives the three phases equal weight, rather than allowing the longest phase to dominate. Allocate the global expert budget across layers from these routing statistics, then retain each layer's selected original expert IDs. The number in the mask name is the mean layer budget, not a uniform count.
- Physically copy selected quantized expert rows from the DASLab IQ2_XS 512-expert source. No expert retraining, implantation, or new requantization is performed. Attention, shared experts, and the large per-layer embedding (PLE) table retain their source bytes.
- Repair router compaction: GGUF stores an
qwen4exp.expert_countarray, original expert IDs,cnm.expert_source_count = 512, andcnm.preserve_router_layout = true. The fork reconstructs source-width routers, masks removed rows before softmax, then maps original IDs to compact expert slots. This avoids the observed router-layout numerical failure; it does not establish exact numerical equivalence to runtime masking.
Related earlier fixed-width build: K192p-implant16 v2. It also passed the PR-6 L3a hidden tests in one local run (REPORT11). Its source and selection process differ; it is not an intermediate parent of this build.
Measurements and sources
All measurements are ours. EVALUATION.json contains per-text KL, warm speed samples, and sanitized task/run identifiers and grades. Raw private task texts and transcripts are not distributed.
Distribution check. KL-H: 0.813 against the full 512-expert DASLab IQ3_S reference on seven recorded ABAP texts. The uncut IQ2_XS 512-expert model scores 0.712 on the same comparison. These are equal-text means; chunk counts differ by text (chunk-weighted mean for this cut: 0.797). This reference differs from KL-L, so their numbers cannot be ranked directly.
Action gate: tool calls on 20/20 repetitions of the first step of one recorded agentic task. This tests willingness to act on that prompt, not correctness or completion of twenty tasks.
Local throughput: 2ร RTX 4060 Ti 16 GB, i7-10700K, 128 GB DDR4, repaired varexp runtime. Target KV q8_0/q4_0, physical batch 384, logical batch 4096, context 194,560, split 27:22; all routed expert layers on GPU.
| Mode | Decode tok/s | PP 8K tok/s | Draft acceptance |
|---|---|---|---|
| MTP + ngram | 67.30 | 967.42 | 67.37% |
| Plain, no draft | 42.26 | 1059.26 | โ |
Medians of three warm samples, after one excluded warm-up per mode/test. Same saved coding-prompt token IDs: decode = 2,048 prompt + 512 generated tokens; prefill = 8,192 prompt + one generated token. Temperature 0, seed 4242, EOS ignored, prefix reuse disabled and reported cache tokens zero. Draft acceptance is the median combined MTP/ngram accepted/generated ratio. Plain removes the MTP head and uses --spec-type none. This short fixed-prompt test is not sustained long-context agent throughput. Source: round10 speed summaries, reproduced in EVALUATION.json.
Agentic checks. The complete ABAP SMOKE v2.3 local series resolved 11/12 tasks on an i7-10700K with two RTX 4060 Ti 16 GB GPUs; only osg-l0-body-003 failed. Package: 573676f648c31ac57b972589168439c31c82e4ec. REPORT12 combines eight newly run tasks (8/8) with the four earlier round10 tasks (3/4), without rerunning those agents. Target KV q8_0/q4_0, physical batch 384, context 194,560, MTP + ngram, Claude Code xhigh, temperature 0.6, and a 5ร time budget. The four-task local subset (SMOKE-4) resolved 3/4: abapgit-l0-line-018, open-abap-apc-l1-method-002, and open-abap-core-l0-body-018 passed; osg-l0-body-003 failed. One stochastic run per task/configuration; the local runs use target KV q8_0/q4_0, physical batch 384 on two 16 GB GPUs and a 5ร time budget. Cloud and local subsets are separate observations.
PR-6 subset on cloud: both L2a and L3a passed hidden tests in the following runs. Harness: Claude Code, xhigh effort, temperature 0.6, context 194,560, target KV q8_0/q5_1, no MTP head, physical batch 2048, 1ร benchmark time budget. These cloud measurements do not use the local speed configuration.
| Task | Hidden grade | Agent seconds | Agent stop |
|---|---|---|---|
| L2a-osg693 | RESOLVED | 2430 | DONE |
| L3a-minz39 | RESOLVED | 3600 | TIMEOUT |
For L3a the agent hit its 3,600-second time limit; its final recorded patch nevertheless passed hidden grading (RESOLVED). Passing hidden tests is distinct from the agent finishing normally. Identifiers and final grades are preserved in EVALUATION.json; no first-green claim is made here.
Known limitation. In our 20-prompt Russian test (one sample per prompt, 400-token cap including reasoning), the ABAP-selected cuts mostly answered and reasoned in English. Cyrillic shares of pooled final-answer letters, excluding code, were PB3-208 2.5%, PB3-304-q2 0%, and K192p 37%, versus 98% for uncut DASLab 512 IQ2_XS. Many finals were empty because reasoning consumed the cap; K192p also had a repetition loop. This small serving-profile comparison does not isolate pruning or establish Russian ability at larger budgets.
Limits: preservation of the full model's pass rate remains unproven. Small, familiar task sets, reused recorded trajectories, and one run per task do not establish generalization or statistical equivalence. KL and action gates are proxies. We do not claim a standard benchmark score, a universal speedup, validated general knowledge, vision quality, other hardware fit, or compatibility with mainline/Ollama. Reduced KV precision is an additional quality tradeoff; short-context checks do not prove quality across 190K contexts.
Serving on two 16 GB GPUs
Build the CUDA fork at the pinned commit above. Download both shards into the same directory, along with the template and MTP head; point the server at shard 1. This is the tested q8_0/q4_0 configuration:
llama-server -m Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-00001-of-00002.gguf \
-c 194560 -np 1 -fa on --jinja \
--chat-template-file chat-template-mid-system.jinja \
-ctk q8_0 -ctv q4_0 -ub 384 -b 4096 -t 8 -tb 16 \
--device CUDA0,CUDA1 -ts 27,22 -ngl 999 --fit off --fit-target 128 \
-md mtp-Qwen3.8-Flash-Next-exps-Q4_K.gguf -devd CUDA1 -otd token_embd=CPU \
--spec-type draft-mtp,ngram-simple --timeout 1800 \
--temp 0.6 --top-k 20 --top-p 0.95 --min-p 0
The 28.8 GB PLE tensor and token embeddings are memory-mapped in system RAM; the GGUF's total size is not its VRAM requirement. All 48 routed expert layers were GPU resident. Memory fit depends on GPU availability and backend; health alone is insufficient, so validate with a real request. The mixed KV pair can produce the existing backend's conversion warnings. For plain decode, remove the three draft-model options and set --spec-type none.
The included 2.984 GB MTP head is byte-identical to v2: unsloth's Q8_0 draft with expert tensors requantized to Q4_K (down to Q5_0). Draft tokens are verified by the main model. The template is also byte-identical to v2 and renders mid-conversation system messages in place, needed by agentic clients. Sampling parameters sent by a client override server defaults.
Files and integrity
Original single-file size: 53,510,748,928 bytes; SHA-256 fe08116b7f806ec50615b621cafd31c1f6fb5ef75e5b12b7efbc11c82f567e87.
| File | Bytes | SHA-256 |
|---|---|---|
Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-00001-of-00002.gguf |
24,710,610,720 | 93903073a081cbec43fd399e2e01e544da2590f24eec44c977ff398cb227e627 |
Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-00002-of-00002.gguf |
28,811,141,152 | df33d7cf39283949a15876a4a9261c7c375663644438f8db43ab10708c11dfe0 |
mtp-Qwen3.8-Flash-Next-exps-Q4_K.gguf |
2,983,995,520 | 5075e39379796203ca59f9248871379d30b4c7062094b44c78d5729406dea406 |
The two standard GGUF shards retain every original metadata value (plus split.*) and all tensor data. A separate original-layout reconstruction, using the preserved header and tensor padding, is required for a byte-identical archive merge; ordinary GGUF mergers may rewrite the header. The archive preparation verifies that reconstruction against the original SHA-256, and independently checks the shards by content. experts-original-ids.json records original IDs (0โ511) by layer. EVALUATION.json records the numerical evidence.
Credits and license
Base weights: Qwen3.8-Flash-Next by Qwen, Qwen Community License 1.0 (LICENSE). Please read the included license for its terms. ISTA-DASLab produced the GSQ-RCO quantization release; its distribution carries Apache-2.0 attribution alongside the underlying Qwen weight license (LICENSE-APACHE-2.0, NOTICE-ISTA-DASLab.md). Credit for the quantization belongs to ISTA-DASLab; this derivative selects expert subsets and implements ragged-layer execution. The MTP head derives from unsloth/Qwen3.8-Flash-Next-GGUF under the Qwen license. The two license files are byte-identical to K192p v2.
- Downloads last month
- 532
We're not able to determine the quantization variants.
Model tree for oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF
Base model
Qwen/Qwen3.8-Flash-Next