Gemma-4-26B-A4B-Weight-Space-Programming (WSP v3.9 — Sparse MoE 4-Way Factorial Ablation)

Training-free, closed-form Multi-Subspace Weight-Space Programming (WSP) for google/gemma-4-26B-A4B-it (Sparse MoE: 26.54B total parameters, ~4B active per token, 30 layers, 128 routed experts + 1 shared dense GeGLU MLP) — zero gradient descent, zero fine-tuning, synthesized in 18.9 seconds on CPU in <90 MB RAM.

Best Configuration — Arm 3 (wsp_gemma4_subspace4_mid.gguf Only): Across the full 123-task 4-Way Factorial Ablation, Arm 3 (Mid-Layer Global Full-Attention + Router-Aligned WSP at Layers 5, 11, 17, 23) achieved the most improvement at 56/123 (45.5%) (+3.2 percentage points over baseline 52/123 = 42.3%), lifting GSM8K 5× (1/20 = 5.0% → 5/20 = 25.0%, +20.0 pts) and HumanEval 2× (1/15 = 6.7% → 2/15 = 13.3%) while preserving 100% TruthfulQA and 93.3% BFCL Irrelevance.


1. 4-Way Factorial Ablation Results (123 Live Benchmark Tasks)

Evaluated via llama-server.exe (gemma-4-26B-A4B-it-UD-Q4_K_M.gguf, ~26 tok/s, greedy decoding temperature=0.0).

Benchmark Suite (Tasks) Arm 1: None (Base) Arm 2: Late-Layer Shared-MLP Only (20–29) Arm 3: Mid-Layer Full-Attn + Router Only (5,11,17,23) — BEST Arm 4: Both Combined (Mid + Late)
Suite A: Shell Execution (8) 3/8 (37.5%) 3/8 (37.5%) 3/8 (37.5%) 3/8 (37.5%)
Suite B: Web Search & Fetch (8) 3/8 (37.5%) 3/8 (37.5%) 3/8 (37.5%) 3/8 (37.5%)
Suite C: Math Reasoning (8) 0/8 (0.0%) 0/8 (0.0%) 1/8 (12.5%) 1/8 (12.5%)
Suite D: Anti-Sycophancy (8) 8/8 (100.0%) 8/8 (100.0%) 8/8 (100.0%) 8/8 (100.0%)
Suite E: Tool Restraint (6) 6/6 (100.0%) 4/6 (66.7%) 4/6 (66.7%) 4/6 (66.7%)
Mobile Agent Subtotal (38) 20/38 (52.6%) 18/38 (47.4%) 19/38 (50.0%) 19/38 (50.0%)
InterCode-NL2Bash (20) 1/20 (5.0%) 1/20 (5.0%) 1/20 (5.0%) 1/20 (5.0%)
BFCL v3 Irrelevance (15) 14/15 (93.3%) 14/15 (93.3%) 14/15 (93.3%) 14/15 (93.3%)
GSM8K Test Split (20) 1/20 (5.0%) 5/20 (25.0%) 5/20 (25.0%) 4/20 (20.0%)
TruthfulQA Generation (15) 15/15 (100.0%) 15/15 (100.0%) 15/15 (100.0%) 15/15 (100.0%)
HumanEval pass@1 (15) 1/15 (6.7%) 2/15 (13.3%) 2/15 (13.3%) 1/15 (6.7%)
Established Subtotal (85) 32/85 (37.6%) 37/85 (43.5%) 37/85 (43.5%) 35/85 (41.2%)
GRAND TOTAL (123) 52/123 (42.3%) 55/123 (44.7%) (+2.4) 56/123 (45.5%) (+3.2 — MOST IMPROVEMENT) 54/123 (43.9%) (+1.6)

2. Sparse MoE Mechanistic Discovery (experts.down_proj vs. mlp.down_proj)

In Gemma4TextDecoderLayer, the Shared Dense MLP (mlp.down_proj, 2112 cols) runs in parallel with 128 Routed Experts (experts.down_proj, 128 × 704 cols), and each branch is independently RMS-normalized (post_feedforward_layernorm_1 and post_feedforward_layernorm_2) before summation.

Direct spectral probing at Layer 29 showed 5×–10× stronger column alignment with formatting-loop ('▁▁▁▁', '▁▁▁', '▁▁▁▁▁▁▁▁', '*') and control tokens (<channel|>, <|tool_call>) inside the 128 Routed Experts than in the Shared Dense MLP:

Layer 29 Tensor 140 ('▁▁▁▁') 139 ('▁▁▁') 144 ('▁▁▁▁▁▁▁▁') 236829 ('*') 101 (<channel|>) 48 (<|tool_call>)
Shared Dense MLP (mlp.down_proj) 0.0333 0.0239 0.0328 0.0304 0.0100 0.0084
128 Routed Experts (experts.down_proj) 0.1250 0.1730 0.1023 0.1831 0.0977 0.0885

Because standard GGUF LoRAs target 2D dense projections (blk.{l}.ffn_down.weight) rather than 3D routed-expert tensors, **Arm 3 (wsp_gemma4_subspace4_mid.gguf)**—which steers the 4 Global Full-Attention anchor layers (5, 11, 17, 23) aligned with top-8 router.proj.weight expert directions—achieves the cleanest and largest intelligence gain (5× GSM8K, 2× HumanEval, +3.2 pts overall).


3. Repository Contents

  • Recommended Best Adapter (Arm 3): wsp_gemma4_subspace4_mid.gguf (1.20 MB, Layers 5, 11, 17, 23, rank 16)
  • Late-Layer Subspace GGUF LoRAs (Layers 20–29, rank 16):
    • wsp_gemma4_subspace1_think.gguf (3.01 MB)
    • wsp_gemma4_subspace2_restraint.gguf (3.01 MB)
    • wsp_gemma4_subspace3_agent.gguf (3.01 MB)
    • wsp_gemma4_combined_late.gguf (4.51 MB, rank 24)
  • Safetensors Delta Patch & Manifest: wsp_gemma4_a4b_patch.safetensors, wsp_gemma4_a4b_manifest.json
  • Reproducible Scripts (scripts/): build_gemma4_a4b_subspaces.py, probe_gemma4_a4b_moe.py, probe_lora_scales.py, run_gemma4_a4b_benchmarks.py, run_gemma4_4way_ablation.py
  • Empirical Benchmark JSONs (benchmarks/): gemma4_a4b_4way_ablation_results.json, base_gemma4_a4b_agent_results.json, base_gemma4_a4b_established_results.json, gemma4_a4b_moe_probe.json

4. Quick Usage with llama-server (Arm 3 — Best Overall)

llama-server -m gemma-4-26B-A4B-it-UD-Q4_K_M.gguf --jinja --lora-scaled wsp_gemma4_subspace4_mid.gguf:2.0

Terminal-Bench 2.0 Evaluation (20 Live Stateful Container Tasks)

Terminal-Bench 2.0 (gemma4_26b_a4b): Base 5/20 (25.0%) -> WSP 6/20 (30.0%) (+5.0 pp; WSP breakdown 4/10 E · 1/9 M · 1/1 H, avg completion tokens 934.4 -> 896.3, 17.8 tok/s, errs=0).
Evaluates best Arm 3 (wsp_gemma4_subspace4_mid.gguf:2.0 on global Full-Attention Layers 5, 11, 17, 23) via llama-server (--cpu-moe -ngl 99).

Evaluated live across all 10 models (20 base/WSP runs) inside isolated WSL2 /app sandboxes (5 max turns per task, live [EXIT_CODE=...] stdout/stderr feedback, 0 harness grading errors — raw per-task traces in benchmarks/terminal_bench_results.json):

Model (slug) Hugging Face Repo WSP Variant Base (20 Tasks) WSP (20 Tasks) Net Delta WSP Breakdown (E / M / H) Avg Tokens (Base → WSP) Speed (tok/s)
qwen35_0_8b Qwen3.5-0.8B-iPhone-Agent-Weight-Space-Programming Dynamic 4S Edge-TB BF16 Hooks 2/20 (10.0%) 2/20 (10.0%) 0.0 pp 2/10 E · 0/9 M · 0/1 H 579.2 → 654.9 32.2
qwen35_2b Qwen3.5-2B-Weight-Space-Programming Dynamic 9S LoRA (Proxy) 3/20 (15.0%) 3/20 (15.0%) 0.0 pp 2/10 E · 1/9 M · 0/1 H 715.8 → 677.7 213.1
qwen35_4b Qwen3.5-4B-Weight-Space-Programming Dynamic 11S LoRA (Hybrid 9S + TB S10/S11) 3/20 (15.0%) 4/20 (20.0%) +5.0 pp 3/10 E · 1/9 M · 0/1 H 604.6 → 650.0 107.7
nemotron3_4b Nemotron-3-Nano-4B-Weight-Space-Programming Dynamic 9S LoRA (BF16 Hooks) 3/20 (15.0%) 3/20 (15.0%) 0.0 pp 3/10 E · 0/9 M · 0/1 H 830.1 → 649.6 118.0
qwen35_9b Qwen3.5-9B-Weight-Space-Programming Dynamic 11S LoRA (BF16 9S + TB S10/S11) 7/20 (35.0%) 7/20 (35.0%) 0.0 pp 5/10 E · 1/9 M · 1/1 H 686.0 → 700.2 68.4
gemma4_e2b gemma-4-E2B-it-iPhone-Agent-Weight-Space-Programming Dynamic 4S Edge-TB BF16 Hooks 4/20 (20.0%) 4/20 (20.0%) 0.0 pp 3/10 E · 0/9 M · 1/1 H 415.2 → 426.6 20.9
gemma4_e4b gemma-4-E4B-it-weight-space-programming Dynamic 4S Edge-TB BF16 Hooks 4/20 (20.0%) 4/20 (20.0%) 0.0 pp 3/10 E · 0/9 M · 1/1 H 723.4 → 498.6 16.4
gemma4_12b Gemma-4-12B-Weight-Space-Programming Dynamic 9S LoRA (BF16 Hooks) 6/20 (30.0%) 6/20 (30.0%) 0.0 pp 5/10 E · 0/9 M · 1/1 H 1010.6 → 928.0 50.7
gemma4_26b_a4b Gemma-4-26B-A4B-Weight-Space-Programming MoE Mid-Attn S4 (scale=2.0) 5/20 (25.0%) 6/20 (30.0%) +5.0 pp 4/10 E · 1/9 M · 1/1 H 934.4 → 896.3 17.8
qwen38_27b Qwen3.8-27B-Weight-Space-Programming Dynamic 4S Mid+Late LoRA 6/20 (30.0%) 8/20 (40.0%) +10.0 pp 6/10 E · 1/9 M · 1/1 H 1013.6 → 1075.3 6.5

Fair-evaluation parity note: qwen35_0_8b stops cleanly on [248046 (<|im_end|>), 248044 (<|endoftext|>)] (eliminating multi-turn <tool_response> hallucination loops where pre-fix Base emitted 2068.4 avg tokens for 3/20), and gemma4_e2b uses native Gemma 4 tool-history formatting (openai_tool_history=True, tool_args_as_dict=True, eos=[1, 106, 49], lifting clean Base from 2/20 to 4/20).

Downloads last month
138
GGUF
Model size
1.18M params
Architecture
gemma4
Hardware compatibility
Log In to add your hardware

We're not able to determine the quantization variants.

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for jstarks/Gemma-4-26B-A4B-Weight-Space-Programming

Adapter
(110)
this model