--- license: apache-2.0 base_model: google/gemma-4-26B-A4B-it tags: - weight-space-programming - mechanistic-interpretability - gemma4 - mixture-of-experts - sparse-moe - gguf - lora - reasoning - gsm8k - humaneval pipeline_tag: text-generation library_name: transformers --- # Gemma-4-26B-A4B-Weight-Space-Programming (WSP v3.9 — Sparse MoE 4-Way Factorial Ablation) **Training-free, closed-form Multi-Subspace Weight-Space Programming (WSP) for [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it) (Sparse MoE: `26.54B` total parameters, `~4B` active per token, `30` layers, `128` routed experts + `1` shared dense GeGLU MLP) — zero gradient descent, zero fine-tuning, synthesized in 18.9 seconds on CPU in `<90 MB` RAM.** > [!IMPORTANT] > **Best Configuration — Arm 3 (`wsp_gemma4_subspace4_mid.gguf` Only):** > Across the full **123-task 4-Way Factorial Ablation**, **Arm 3 (Mid-Layer Global Full-Attention + Router-Aligned WSP at Layers 5, 11, 17, 23)** achieved the **most improvement** at **`56/123 (45.5%)`** (**+3.2 percentage points** over baseline `52/123 = 42.3%`), lifting **GSM8K 5× (`1/20 = 5.0%` → `5/20 = 25.0%`, +20.0 pts)** and **HumanEval 2× (`1/15 = 6.7%` → `2/15 = 13.3%`)** while preserving `100%` TruthfulQA and `93.3%` BFCL Irrelevance. --- ## 1. 4-Way Factorial Ablation Results (123 Live Benchmark Tasks) Evaluated via `llama-server.exe` (`gemma-4-26B-A4B-it-UD-Q4_K_M.gguf`, `~26 tok/s`, greedy decoding `temperature=0.0`). | Benchmark Suite (Tasks) | Arm 1: None (Base) | Arm 2: Late-Layer Shared-MLP Only (`20–29`) | **Arm 3: Mid-Layer Full-Attn + Router Only (`5,11,17,23`) — BEST** | Arm 4: Both Combined (`Mid + Late`) | | :--- | :---: | :---: | :---: | :---: | | **Suite A: Shell Execution (8)** | `3/8 (37.5%)` | `3/8 (37.5%)` | `3/8 (37.5%)` | `3/8 (37.5%)` | | **Suite B: Web Search & Fetch (8)** | `3/8 (37.5%)` | `3/8 (37.5%)` | `3/8 (37.5%)` | `3/8 (37.5%)` | | **Suite C: Math Reasoning (8)** | `0/8 (0.0%)` | `0/8 (0.0%)` | **`1/8 (12.5%)`** | **`1/8 (12.5%)`** | | **Suite D: Anti-Sycophancy (8)** | `8/8 (100.0%)` | `8/8 (100.0%)` | `8/8 (100.0%)` | `8/8 (100.0%)` | | **Suite E: Tool Restraint (6)** | `6/6 (100.0%)` | `4/6 (66.7%)` | `4/6 (66.7%)` | `4/6 (66.7%)` | | **Mobile Agent Subtotal (38)** | **`20/38 (52.6%)`** | `18/38 (47.4%)` | `19/38 (50.0%)` | `19/38 (50.0%)` | | **InterCode-NL2Bash (20)** | `1/20 (5.0%)` | `1/20 (5.0%)` | `1/20 (5.0%)` | `1/20 (5.0%)` | | **BFCL v3 Irrelevance (15)** | `14/15 (93.3%)` | `14/15 (93.3%)` | `14/15 (93.3%)` | `14/15 (93.3%)` | | **GSM8K Test Split (20)** | `1/20 (5.0%)` | **`5/20 (25.0%)`** | **`5/20 (25.0%)`** | `4/20 (20.0%)` | | **TruthfulQA Generation (15)** | `15/15 (100.0%)` | `15/15 (100.0%)` | `15/15 (100.0%)` | `15/15 (100.0%)` | | **HumanEval pass@1 (15)** | `1/15 (6.7%)` | **`2/15 (13.3%)`** | **`2/15 (13.3%)`** | `1/15 (6.7%)` | | **Established Subtotal (85)** | `32/85 (37.6%)` | **`37/85 (43.5%)`** | **`37/85 (43.5%)`** | `35/85 (41.2%)` | | **GRAND TOTAL (123)** | **`52/123 (42.3%)`** | **`55/123 (44.7%)` (+2.4)** | **`56/123 (45.5%)` (+3.2 — MOST IMPROVEMENT)** | **`54/123 (43.9%)` (+1.6)** | --- ## 2. Sparse MoE Mechanistic Discovery (`experts.down_proj` vs. `mlp.down_proj`) In `Gemma4TextDecoderLayer`, the Shared Dense MLP (`mlp.down_proj`, `2112` cols) runs in parallel with 128 Routed Experts (`experts.down_proj`, `128 × 704` cols), and each branch is independently RMS-normalized (`post_feedforward_layernorm_1` and `post_feedforward_layernorm_2`) before summation. Direct spectral probing at Layer 29 showed **5×–10× stronger column alignment** with formatting-loop (`'▁▁▁▁'`, `'▁▁▁'`, `'▁▁▁▁▁▁▁▁'`, `'*'`) and control tokens (``, `<|tool_call>`) inside the **128 Routed Experts** than in the Shared Dense MLP: | Layer 29 Tensor | `140` (`'▁▁▁▁'`) | `139` (`'▁▁▁'`) | `144` (`'▁▁▁▁▁▁▁▁'`) | `236829` (`'*'`) | `101` (``) | `48` (`<\|tool_call>`) | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | **Shared Dense MLP (`mlp.down_proj`)** | `0.0333` | `0.0239` | `0.0328` | `0.0304` | `0.0100` | `0.0084` | | **128 Routed Experts (`experts.down_proj`)** | **`0.1250`** | **`0.1730`** | **`0.1023`** | **`0.1831`** | **`0.0977`** | **`0.0885`** | Because standard GGUF LoRAs target 2D dense projections (`blk.{l}.ffn_down.weight`) rather than 3D routed-expert tensors, **Arm 3 (`wsp_gemma4_subspace4_mid.gguf`)**—which steers the 4 Global Full-Attention anchor layers (`5, 11, 17, 23`) aligned with top-8 `router.proj.weight` expert directions—achieves the cleanest and largest intelligence gain (**5× GSM8K**, **2× HumanEval**, **+3.2 pts overall**). --- ## 3. Repository Contents - **Recommended Best Adapter (Arm 3):** `wsp_gemma4_subspace4_mid.gguf` (`1.20 MB`, Layers `5, 11, 17, 23`, rank 16) - **Late-Layer Subspace GGUF LoRAs (Layers `20–29`, rank 16):** - `wsp_gemma4_subspace1_think.gguf` (`3.01 MB`) - `wsp_gemma4_subspace2_restraint.gguf` (`3.01 MB`) - `wsp_gemma4_subspace3_agent.gguf` (`3.01 MB`) - `wsp_gemma4_combined_late.gguf` (`4.51 MB`, rank 24) - **Safetensors Delta Patch & Manifest:** `wsp_gemma4_a4b_patch.safetensors`, `wsp_gemma4_a4b_manifest.json` - **Reproducible Scripts (`scripts/`):** `build_gemma4_a4b_subspaces.py`, `probe_gemma4_a4b_moe.py`, `probe_lora_scales.py`, `run_gemma4_a4b_benchmarks.py`, `run_gemma4_4way_ablation.py` - **Empirical Benchmark JSONs (`benchmarks/`):** `gemma4_a4b_4way_ablation_results.json`, `base_gemma4_a4b_agent_results.json`, `base_gemma4_a4b_established_results.json`, `gemma4_a4b_moe_probe.json` ## 4. Quick Usage with `llama-server` (Arm 3 — Best Overall) ```bash llama-server -m gemma-4-26B-A4B-it-UD-Q4_K_M.gguf --jinja --lora-scaled wsp_gemma4_subspace4_mid.gguf:2.0 ``` --- ## Terminal-Bench 2.0 Evaluation (`20` Live Stateful Container Tasks) > [!NOTE] > **Terminal-Bench 2.0 (`gemma4_26b_a4b`):** Base `5/20 (25.0%)` -> **WSP `6/20 (30.0%)`** (**`+5.0 pp`**; WSP breakdown `4/10 E · 1/9 M · 1/1 H`, avg completion tokens `934.4 -> 896.3`, `17.8 tok/s`, `errs=0`). > Evaluates best Arm 3 (`wsp_gemma4_subspace4_mid.gguf:2.0` on global Full-Attention Layers 5, 11, 17, 23) via `llama-server` (`--cpu-moe -ngl 99`). Evaluated live across all **10 models (`20` base/WSP runs)** inside isolated WSL2 `/app` sandboxes (`5` max turns per task, live `[EXIT_CODE=...]` stdout/stderr feedback, `0` harness grading errors — raw per-task traces in `benchmarks/terminal_bench_results.json`): | Model (`slug`) | Hugging Face Repo | WSP Variant | Base (`20` Tasks) | **WSP (`20` Tasks)** | Net Delta | WSP Breakdown (`E / M / H`) | Avg Tokens (`Base → WSP`) | Speed (`tok/s`) | |---|---|---|---|---|---|---|---|---| | **`qwen35_0_8b`** | [`Qwen3.5-0.8B-iPhone-Agent-Weight-Space-Programming`](https://huggingface.co/jstarks/Qwen3.5-0.8B-iPhone-Agent-Weight-Space-Programming) | Dynamic 4S Edge-TB BF16 Hooks | `2/20 (10.0%)` | **`2/20 (10.0%)`** | **`0.0 pp`** | `2/10 E · 0/9 M · 0/1 H` | `579.2 → 654.9` | `32.2` | | **`qwen35_2b`** | [`Qwen3.5-2B-Weight-Space-Programming`](https://huggingface.co/jstarks/Qwen3.5-2B-Weight-Space-Programming) | Dynamic 9S LoRA (Proxy) | `3/20 (15.0%)` | **`3/20 (15.0%)`** | **`0.0 pp`** | `2/10 E · 1/9 M · 0/1 H` | `715.8 → 677.7` | `213.1` | | **`qwen35_4b`** | [`Qwen3.5-4B-Weight-Space-Programming`](https://huggingface.co/jstarks/Qwen3.5-4B-Weight-Space-Programming) | Dynamic 11S LoRA (Hybrid 9S + TB S10/S11) | `3/20 (15.0%)` | **`4/20 (20.0%)`** | **`+5.0 pp`** | `3/10 E · 1/9 M · 0/1 H` | `604.6 → 650.0` | `107.7` | | **`nemotron3_4b`** | [`Nemotron-3-Nano-4B-Weight-Space-Programming`](https://huggingface.co/jstarks/Nemotron-3-Nano-4B-Weight-Space-Programming) | Dynamic 9S LoRA (BF16 Hooks) | `3/20 (15.0%)` | **`3/20 (15.0%)`** | **`0.0 pp`** | `3/10 E · 0/9 M · 0/1 H` | `830.1 → 649.6` | `118.0` | | **`qwen35_9b`** | [`Qwen3.5-9B-Weight-Space-Programming`](https://huggingface.co/jstarks/Qwen3.5-9B-Weight-Space-Programming) | Dynamic 11S LoRA (BF16 9S + TB S10/S11) | `7/20 (35.0%)` | **`7/20 (35.0%)`** | **`0.0 pp`** | `5/10 E · 1/9 M · 1/1 H` | `686.0 → 700.2` | `68.4` | | **`gemma4_e2b`** | [`gemma-4-E2B-it-iPhone-Agent-Weight-Space-Programming`](https://huggingface.co/jstarks/gemma-4-E2B-it-iPhone-Agent-Weight-Space-Programming) | Dynamic 4S Edge-TB BF16 Hooks | `4/20 (20.0%)` | **`4/20 (20.0%)`** | **`0.0 pp`** | `3/10 E · 0/9 M · 1/1 H` | `415.2 → 426.6` | `20.9` | | **`gemma4_e4b`** | [`gemma-4-E4B-it-weight-space-programming`](https://huggingface.co/jstarks/gemma-4-E4B-it-weight-space-programming) | Dynamic 4S Edge-TB BF16 Hooks | `4/20 (20.0%)` | **`4/20 (20.0%)`** | **`0.0 pp`** | `3/10 E · 0/9 M · 1/1 H` | `723.4 → 498.6` | `16.4` | | **`gemma4_12b`** | [`Gemma-4-12B-Weight-Space-Programming`](https://huggingface.co/jstarks/Gemma-4-12B-Weight-Space-Programming) | Dynamic 9S LoRA (BF16 Hooks) | `6/20 (30.0%)` | **`6/20 (30.0%)`** | **`0.0 pp`** | `5/10 E · 0/9 M · 1/1 H` | `1010.6 → 928.0` | `50.7` | | **`gemma4_26b_a4b`** | [`Gemma-4-26B-A4B-Weight-Space-Programming`](https://huggingface.co/jstarks/Gemma-4-26B-A4B-Weight-Space-Programming) | MoE Mid-Attn S4 (`scale=2.0`) | `5/20 (25.0%)` | **`6/20 (30.0%)`** | **`+5.0 pp`** | `4/10 E · 1/9 M · 1/1 H` | `934.4 → 896.3` | `17.8` | | **`qwen38_27b`** | [`Qwen3.8-27B-Weight-Space-Programming`](https://huggingface.co/jstarks/Qwen3.8-27B-Weight-Space-Programming) | Dynamic 4S Mid+Late LoRA | `6/20 (30.0%)` | **`8/20 (40.0%)`** | **`+10.0 pp`** | `6/10 E · 1/9 M · 1/1 H` | `1013.6 → 1075.3` | `6.5` | > *Fair-evaluation parity note:* `qwen35_0_8b` stops cleanly on `[248046 (<|im_end|>), 248044 (<|endoftext|>)]` (eliminating multi-turn `` hallucination loops where pre-fix Base emitted `2068.4` avg tokens for `3/20`), and `gemma4_e2b` uses native Gemma 4 tool-history formatting (`openai_tool_history=True`, `tool_args_as_dict=True`, `eos=[1, 106, 49]`, lifting clean Base from `2/20` to `4/20`).