Qwen3.8-Flash-Next ยท GSQ-RCO ยท MoREL PB3-304-q2 ยท GGUF

Requires our llama.cpp fork, branch cnm/varexp, commit 7ed1a0a57e5621073146bca8d5c166e52cfe462b. Mainline llama.cpp will not load this model.

MoREL = Mixture of Ragged Experts per Layer: a physically pruned MoE with a different expert count in each layer. This 2-bit build is cut from the 512-expert IQ2_XS variant of ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF, using the PB3-304 mask. The mask keeps a mean of 304 experts per layer (range 234โ€“364, 48 routed layers), with top-10 routing unchanged. The quantization label describes the source release; individual tensor types are mixed. This is an independent community derivative, without endorsement from Qwen or ISTA-DASLab.

Downloads / prebuilt binaries

MoREL runtime cnm-varexp-v0.1 prerelease provides the fork runtime required by the PB3 MoREL models. It is also offered here for K192p-implant16 v2, whose generation with these packages has not been tested. These are machine-specific Ubuntu 24.04 packages. Choose the package that matches your CPU and backend; they are not universal Linux builds. Windows x64 CUDA/CPU, macOS arm64 and generic Linux x64 CUDA/Vulkan/CPU Actions packages are not available in this prerelease yet. Build from the release source for other machines.

Machine / backend Asset SHA-256
AVX2/FMA/F16C CPU (i10, Ryzen 5 5600G) llama-cnm-varexp-v0.1-i10-cpu-linux-x86_64.tar.gz 8c40381780f37427c52d7a5b30efe07bf0e0448ad0880f1a7d025ebf2ad0876e
AVX CPU + Radeon Vulkan (i3, i5-3570K / RX 580) llama-cnm-varexp-v0.1-i3-vulkan-linux-x86_64.tar.gz 5420ec8769075c1affe3b011b5a0d7dc92fa3cd688accfb7b8ee4b6e5b07f345
CUDA 12.0 sm_75 (i5, RTX 2070) llama-cnm-varexp-v0.1-i5-cuda-linux-x86_64.tar.gz 7c7390f8193197fb38b8e3aa9ddd5507afa6caa21aaed19672e4c9857a8c502c
CUDA 12.0 sm_89/86/75 (i7, RTX 4060 Ti) llama-cnm-varexp-v0.1-i7-cuda-linux-x86_64.tar.gz 567d67ff6ca26c3046c39247b8efa704f897d981f2fa2671a6edd7ebe8a7e16b

Each tarball has a matching .sha256 asset on the release page. Download both and run sha256sum -c <archive>.sha256 before extraction. Keep the executables and project shared libraries together. System C/C++ libraries and GPU drivers/CUDA libraries are external dependencies. These packages disable HTTPS and the embedded server Web UI.

The release notes contain the complete test matrix. Packaging checks covered all four builds; PB3-208 had short CPU (i10) and partial-offload Vulkan (i3) completions. CUDA generation, PB3-304/K192p generation with these packaged binaries, full Vulkan residency, long contexts and MTP were not tested in this release. Existing model-card evaluations used their separately recorded runtime configurations.

How it was made

  1. Record expert-routing tomography while an agent works through ABAP coding sessions, including trajectories produced by other models. Separate explore, edit, and verify phases.
  2. PB3 gives the three phases equal weight, rather than allowing the longest phase to dominate. Allocate the global expert budget across layers from these routing statistics, then retain each layer's selected original expert IDs. The number in the mask name is the mean layer budget, not a uniform count.
  3. Physically copy selected quantized expert rows from the DASLab IQ2_XS 512-expert source. No expert retraining, implantation, or new requantization is performed. Attention, shared experts, and the large per-layer embedding (PLE) table retain their source bytes.
  4. Repair router compaction: GGUF stores an qwen4exp.expert_count array, original expert IDs, cnm.expert_source_count = 512, and cnm.preserve_router_layout = true. The fork reconstructs source-width routers, masks removed rows before softmax, then maps original IDs to compact expert slots. This avoids the observed router-layout numerical failure; it does not establish exact numerical equivalence to runtime masking.

Related earlier fixed-width build: K192p-implant16 v2. It also passed the PR-6 L3a hidden tests in one local run (REPORT11). Its source and selection process differ; it is not an intermediate parent of this build.

Measurements and sources

All measurements are ours. EVALUATION.json contains per-text KL, warm speed samples, and sanitized task/run identifiers and grades. Raw private task texts and transcripts are not distributed.

Distribution check. KL-H: 0.813 against the full 512-expert DASLab IQ3_S reference on seven recorded ABAP texts. The uncut IQ2_XS 512-expert model scores 0.712 on the same comparison. These are equal-text means; chunk counts differ by text (chunk-weighted mean for this cut: 0.797). This reference differs from KL-L, so their numbers cannot be ranked directly.

Action gate: tool calls on 20/20 repetitions of the first step of one recorded agentic task. This tests willingness to act on that prompt, not correctness or completion of twenty tasks.

Local throughput: 2ร— RTX 4060 Ti 16 GB, i7-10700K, 128 GB DDR4, repaired varexp runtime. Target KV q8_0/q4_0, physical batch 384, logical batch 4096, context 194,560, split 27:22; all routed expert layers on GPU.

Mode Decode tok/s PP 8K tok/s Draft acceptance
MTP + ngram 67.30 967.42 67.37%
Plain, no draft 42.26 1059.26 โ€”

Medians of three warm samples, after one excluded warm-up per mode/test. Same saved coding-prompt token IDs: decode = 2,048 prompt + 512 generated tokens; prefill = 8,192 prompt + one generated token. Temperature 0, seed 4242, EOS ignored, prefix reuse disabled and reported cache tokens zero. Draft acceptance is the median combined MTP/ngram accepted/generated ratio. Plain removes the MTP head and uses --spec-type none. This short fixed-prompt test is not sustained long-context agent throughput. Source: round10 speed summaries, reproduced in EVALUATION.json.

Agentic checks. The complete ABAP SMOKE v2.3 local series resolved 11/12 tasks on an i7-10700K with two RTX 4060 Ti 16 GB GPUs; only osg-l0-body-003 failed. Package: 573676f648c31ac57b972589168439c31c82e4ec. REPORT12 combines eight newly run tasks (8/8) with the four earlier round10 tasks (3/4), without rerunning those agents. Target KV q8_0/q4_0, physical batch 384, context 194,560, MTP + ngram, Claude Code xhigh, temperature 0.6, and a 5ร— time budget. The four-task local subset (SMOKE-4) resolved 3/4: abapgit-l0-line-018, open-abap-apc-l1-method-002, and open-abap-core-l0-body-018 passed; osg-l0-body-003 failed. One stochastic run per task/configuration; the local runs use target KV q8_0/q4_0, physical batch 384 on two 16 GB GPUs and a 5ร— time budget. Cloud and local subsets are separate observations.

PR-6 subset on cloud: both L2a and L3a passed hidden tests in the following runs. Harness: Claude Code, xhigh effort, temperature 0.6, context 194,560, target KV q8_0/q5_1, no MTP head, physical batch 2048, 1ร— benchmark time budget. These cloud measurements do not use the local speed configuration.

Task Hidden grade Agent seconds Agent stop
L2a-osg693 RESOLVED 2430 DONE
L3a-minz39 RESOLVED 3600 TIMEOUT

For L3a the agent hit its 3,600-second time limit; its final recorded patch nevertheless passed hidden grading (RESOLVED). Passing hidden tests is distinct from the agent finishing normally. Identifiers and final grades are preserved in EVALUATION.json; no first-green claim is made here.

Known limitation. In our 20-prompt Russian test (one sample per prompt, 400-token cap including reasoning), the ABAP-selected cuts mostly answered and reasoned in English. Cyrillic shares of pooled final-answer letters, excluding code, were PB3-208 2.5%, PB3-304-q2 0%, and K192p 37%, versus 98% for uncut DASLab 512 IQ2_XS. Many finals were empty because reasoning consumed the cap; K192p also had a repetition loop. This small serving-profile comparison does not isolate pruning or establish Russian ability at larger budgets.

Limits: preservation of the full model's pass rate remains unproven. Small, familiar task sets, reused recorded trajectories, and one run per task do not establish generalization or statistical equivalence. KL and action gates are proxies. We do not claim a standard benchmark score, a universal speedup, validated general knowledge, vision quality, other hardware fit, or compatibility with mainline/Ollama. Reduced KV precision is an additional quality tradeoff; short-context checks do not prove quality across 190K contexts.

Serving on two 16 GB GPUs

Build the CUDA fork at the pinned commit above. Download both shards into the same directory, along with the template and MTP head; point the server at shard 1. This is the tested q8_0/q4_0 configuration:

llama-server -m Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-00001-of-00002.gguf \
  -c 194560 -np 1 -fa on --jinja \
  --chat-template-file chat-template-mid-system.jinja \
  -ctk q8_0 -ctv q4_0 -ub 384 -b 4096 -t 8 -tb 16 \
  --device CUDA0,CUDA1 -ts 27,22 -ngl 999 --fit off --fit-target 128 \
  -md mtp-Qwen3.8-Flash-Next-exps-Q4_K.gguf -devd CUDA1 -otd token_embd=CPU \
  --spec-type draft-mtp,ngram-simple --timeout 1800 \
  --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0

The 28.8 GB PLE tensor and token embeddings are memory-mapped in system RAM; the GGUF's total size is not its VRAM requirement. All 48 routed expert layers were GPU resident. Memory fit depends on GPU availability and backend; health alone is insufficient, so validate with a real request. The mixed KV pair can produce the existing backend's conversion warnings. For plain decode, remove the three draft-model options and set --spec-type none.

The included 2.984 GB MTP head is byte-identical to v2: unsloth's Q8_0 draft with expert tensors requantized to Q4_K (down to Q5_0). Draft tokens are verified by the main model. The template is also byte-identical to v2 and renders mid-conversation system messages in place, needed by agentic clients. Sampling parameters sent by a client override server defaults.

Files and integrity

Original single-file size: 53,510,748,928 bytes; SHA-256 fe08116b7f806ec50615b621cafd31c1f6fb5ef75e5b12b7efbc11c82f567e87.

File Bytes SHA-256
Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-00001-of-00002.gguf 24,710,610,720 93903073a081cbec43fd399e2e01e544da2590f24eec44c977ff398cb227e627
Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-00002-of-00002.gguf 28,811,141,152 df33d7cf39283949a15876a4a9261c7c375663644438f8db43ab10708c11dfe0
mtp-Qwen3.8-Flash-Next-exps-Q4_K.gguf 2,983,995,520 5075e39379796203ca59f9248871379d30b4c7062094b44c78d5729406dea406

The two standard GGUF shards retain every original metadata value (plus split.*) and all tensor data. A separate original-layout reconstruction, using the preserved header and tensor padding, is required for a byte-identical archive merge; ordinary GGUF mergers may rewrite the header. The archive preparation verifies that reconstruction against the original SHA-256, and independently checks the shards by content. experts-original-ids.json records original IDs (0โ€“511) by layer. EVALUATION.json records the numerical evidence.

Credits and license

Base weights: Qwen3.8-Flash-Next by Qwen, Qwen Community License 1.0 (LICENSE). Please read the included license for its terms. ISTA-DASLab produced the GSQ-RCO quantization release; its distribution carries Apache-2.0 attribution alongside the underlying Qwen weight license (LICENSE-APACHE-2.0, NOTICE-ISTA-DASLab.md). Credit for the quantization belongs to ISTA-DASLab; this derivative selects expert subsets and implements ragged-layer execution. The MTP head derives from unsloth/Qwen3.8-Flash-Next-GGUF under the Qwen license. The two license files are byte-identical to K192p v2.

Downloads last month
532
GGUF
Model size
128B params
Architecture
qwen4exp
Hardware compatibility
Log In to add your hardware

We're not able to determine the quantization variants.

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for oisee/Qwen3.8-Flash-Next-GSQ-RCO-IQ2-MoREL-PB3-304-GGUF

Quantized
(54)
this model