jstarks's picture
Update verified 10-model Terminal-Bench 2.0 results (gemma4_26b_a4b) and model card
fad518b verified
|
Raw History Blame Contribute Delete
10.3 kB
metadata
license: apache-2.0
base_model: google/gemma-4-26B-A4B-it
tags:
  - weight-space-programming
  - mechanistic-interpretability
  - gemma4
  - mixture-of-experts
  - sparse-moe
  - gguf
  - lora
  - reasoning
  - gsm8k
  - humaneval
pipeline_tag: text-generation
library_name: transformers

Gemma-4-26B-A4B-Weight-Space-Programming (WSP v3.9 β€” Sparse MoE 4-Way Factorial Ablation)

Training-free, closed-form Multi-Subspace Weight-Space Programming (WSP) for google/gemma-4-26B-A4B-it (Sparse MoE: 26.54B total parameters, ~4B active per token, 30 layers, 128 routed experts + 1 shared dense GeGLU MLP) β€” zero gradient descent, zero fine-tuning, synthesized in 18.9 seconds on CPU in <90 MB RAM.

Best Configuration β€” Arm 3 (wsp_gemma4_subspace4_mid.gguf Only): Across the full 123-task 4-Way Factorial Ablation, Arm 3 (Mid-Layer Global Full-Attention + Router-Aligned WSP at Layers 5, 11, 17, 23) achieved the most improvement at 56/123 (45.5%) (+3.2 percentage points over baseline 52/123 = 42.3%), lifting GSM8K 5Γ— (1/20 = 5.0% β†’ 5/20 = 25.0%, +20.0 pts) and HumanEval 2Γ— (1/15 = 6.7% β†’ 2/15 = 13.3%) while preserving 100% TruthfulQA and 93.3% BFCL Irrelevance.


1. 4-Way Factorial Ablation Results (123 Live Benchmark Tasks)

Evaluated via llama-server.exe (gemma-4-26B-A4B-it-UD-Q4_K_M.gguf, ~26 tok/s, greedy decoding temperature=0.0).

Benchmark Suite (Tasks) Arm 1: None (Base) Arm 2: Late-Layer Shared-MLP Only (20–29) Arm 3: Mid-Layer Full-Attn + Router Only (5,11,17,23) β€” BEST Arm 4: Both Combined (Mid + Late)
Suite A: Shell Execution (8) 3/8 (37.5%) 3/8 (37.5%) 3/8 (37.5%) 3/8 (37.5%)
Suite B: Web Search & Fetch (8) 3/8 (37.5%) 3/8 (37.5%) 3/8 (37.5%) 3/8 (37.5%)
Suite C: Math Reasoning (8) 0/8 (0.0%) 0/8 (0.0%) 1/8 (12.5%) 1/8 (12.5%)
Suite D: Anti-Sycophancy (8) 8/8 (100.0%) 8/8 (100.0%) 8/8 (100.0%) 8/8 (100.0%)
Suite E: Tool Restraint (6) 6/6 (100.0%) 4/6 (66.7%) 4/6 (66.7%) 4/6 (66.7%)
Mobile Agent Subtotal (38) 20/38 (52.6%) 18/38 (47.4%) 19/38 (50.0%) 19/38 (50.0%)
InterCode-NL2Bash (20) 1/20 (5.0%) 1/20 (5.0%) 1/20 (5.0%) 1/20 (5.0%)
BFCL v3 Irrelevance (15) 14/15 (93.3%) 14/15 (93.3%) 14/15 (93.3%) 14/15 (93.3%)
GSM8K Test Split (20) 1/20 (5.0%) 5/20 (25.0%) 5/20 (25.0%) 4/20 (20.0%)
TruthfulQA Generation (15) 15/15 (100.0%) 15/15 (100.0%) 15/15 (100.0%) 15/15 (100.0%)
HumanEval pass@1 (15) 1/15 (6.7%) 2/15 (13.3%) 2/15 (13.3%) 1/15 (6.7%)
Established Subtotal (85) 32/85 (37.6%) 37/85 (43.5%) 37/85 (43.5%) 35/85 (41.2%)
GRAND TOTAL (123) 52/123 (42.3%) 55/123 (44.7%) (+2.4) 56/123 (45.5%) (+3.2 β€” MOST IMPROVEMENT) 54/123 (43.9%) (+1.6)

2. Sparse MoE Mechanistic Discovery (experts.down_proj vs. mlp.down_proj)

In Gemma4TextDecoderLayer, the Shared Dense MLP (mlp.down_proj, 2112 cols) runs in parallel with 128 Routed Experts (experts.down_proj, 128 Γ— 704 cols), and each branch is independently RMS-normalized (post_feedforward_layernorm_1 and post_feedforward_layernorm_2) before summation.

Direct spectral probing at Layer 29 showed 5×–10Γ— stronger column alignment with formatting-loop ('▁▁▁▁', '▁▁▁', '▁▁▁▁▁▁▁▁', '*') and control tokens (<channel|>, <|tool_call>) inside the 128 Routed Experts than in the Shared Dense MLP:

Layer 29 Tensor 140 ('▁▁▁▁') 139 ('▁▁▁') 144 ('▁▁▁▁▁▁▁▁') 236829 ('*') 101 (<channel|>) 48 (<|tool_call>)
Shared Dense MLP (mlp.down_proj) 0.0333 0.0239 0.0328 0.0304 0.0100 0.0084
128 Routed Experts (experts.down_proj) 0.1250 0.1730 0.1023 0.1831 0.0977 0.0885

Because standard GGUF LoRAs target 2D dense projections (blk.{l}.ffn_down.weight) rather than 3D routed-expert tensors, **Arm 3 (wsp_gemma4_subspace4_mid.gguf)**β€”which steers the 4 Global Full-Attention anchor layers (5, 11, 17, 23) aligned with top-8 router.proj.weight expert directionsβ€”achieves the cleanest and largest intelligence gain (5Γ— GSM8K, 2Γ— HumanEval, +3.2 pts overall).


3. Repository Contents

  • Recommended Best Adapter (Arm 3): wsp_gemma4_subspace4_mid.gguf (1.20 MB, Layers 5, 11, 17, 23, rank 16)
  • Late-Layer Subspace GGUF LoRAs (Layers 20–29, rank 16):
    • wsp_gemma4_subspace1_think.gguf (3.01 MB)
    • wsp_gemma4_subspace2_restraint.gguf (3.01 MB)
    • wsp_gemma4_subspace3_agent.gguf (3.01 MB)
    • wsp_gemma4_combined_late.gguf (4.51 MB, rank 24)
  • Safetensors Delta Patch & Manifest: wsp_gemma4_a4b_patch.safetensors, wsp_gemma4_a4b_manifest.json
  • Reproducible Scripts (scripts/): build_gemma4_a4b_subspaces.py, probe_gemma4_a4b_moe.py, probe_lora_scales.py, run_gemma4_a4b_benchmarks.py, run_gemma4_4way_ablation.py
  • Empirical Benchmark JSONs (benchmarks/): gemma4_a4b_4way_ablation_results.json, base_gemma4_a4b_agent_results.json, base_gemma4_a4b_established_results.json, gemma4_a4b_moe_probe.json

4. Quick Usage with llama-server (Arm 3 β€” Best Overall)

llama-server -m gemma-4-26B-A4B-it-UD-Q4_K_M.gguf --jinja --lora-scaled wsp_gemma4_subspace4_mid.gguf:2.0

Terminal-Bench 2.0 Evaluation (20 Live Stateful Container Tasks)

Terminal-Bench 2.0 (gemma4_26b_a4b): Base 5/20 (25.0%) -> WSP 6/20 (30.0%) (+5.0 pp; WSP breakdown 4/10 E Β· 1/9 M Β· 1/1 H, avg completion tokens 934.4 -> 896.3, 17.8 tok/s, errs=0).
Evaluates best Arm 3 (wsp_gemma4_subspace4_mid.gguf:2.0 on global Full-Attention Layers 5, 11, 17, 23) via llama-server (--cpu-moe -ngl 99).

Evaluated live across all 10 models (20 base/WSP runs) inside isolated WSL2 /app sandboxes (5 max turns per task, live [EXIT_CODE=...] stdout/stderr feedback, 0 harness grading errors β€” raw per-task traces in benchmarks/terminal_bench_results.json):

Model (slug) Hugging Face Repo WSP Variant Base (20 Tasks) WSP (20 Tasks) Net Delta WSP Breakdown (E / M / H) Avg Tokens (Base β†’ WSP) Speed (tok/s)
qwen35_0_8b Qwen3.5-0.8B-iPhone-Agent-Weight-Space-Programming Dynamic 4S Edge-TB BF16 Hooks 2/20 (10.0%) 2/20 (10.0%) 0.0 pp 2/10 E Β· 0/9 M Β· 0/1 H 579.2 β†’ 654.9 32.2
qwen35_2b Qwen3.5-2B-Weight-Space-Programming Dynamic 9S LoRA (Proxy) 3/20 (15.0%) 3/20 (15.0%) 0.0 pp 2/10 E Β· 1/9 M Β· 0/1 H 715.8 β†’ 677.7 213.1
qwen35_4b Qwen3.5-4B-Weight-Space-Programming Dynamic 11S LoRA (Hybrid 9S + TB S10/S11) 3/20 (15.0%) 4/20 (20.0%) +5.0 pp 3/10 E Β· 1/9 M Β· 0/1 H 604.6 β†’ 650.0 107.7
nemotron3_4b Nemotron-3-Nano-4B-Weight-Space-Programming Dynamic 9S LoRA (BF16 Hooks) 3/20 (15.0%) 3/20 (15.0%) 0.0 pp 3/10 E Β· 0/9 M Β· 0/1 H 830.1 β†’ 649.6 118.0
qwen35_9b Qwen3.5-9B-Weight-Space-Programming Dynamic 11S LoRA (BF16 9S + TB S10/S11) 7/20 (35.0%) 7/20 (35.0%) 0.0 pp 5/10 E Β· 1/9 M Β· 1/1 H 686.0 β†’ 700.2 68.4
gemma4_e2b gemma-4-E2B-it-iPhone-Agent-Weight-Space-Programming Dynamic 4S Edge-TB BF16 Hooks 4/20 (20.0%) 4/20 (20.0%) 0.0 pp 3/10 E Β· 0/9 M Β· 1/1 H 415.2 β†’ 426.6 20.9
gemma4_e4b gemma-4-E4B-it-weight-space-programming Dynamic 4S Edge-TB BF16 Hooks 4/20 (20.0%) 4/20 (20.0%) 0.0 pp 3/10 E Β· 0/9 M Β· 1/1 H 723.4 β†’ 498.6 16.4
gemma4_12b Gemma-4-12B-Weight-Space-Programming Dynamic 9S LoRA (BF16 Hooks) 6/20 (30.0%) 6/20 (30.0%) 0.0 pp 5/10 E Β· 0/9 M Β· 1/1 H 1010.6 β†’ 928.0 50.7
gemma4_26b_a4b Gemma-4-26B-A4B-Weight-Space-Programming MoE Mid-Attn S4 (scale=2.0) 5/20 (25.0%) 6/20 (30.0%) +5.0 pp 4/10 E Β· 1/9 M Β· 1/1 H 934.4 β†’ 896.3 17.8
qwen38_27b Qwen3.8-27B-Weight-Space-Programming Dynamic 4S Mid+Late LoRA 6/20 (30.0%) 8/20 (40.0%) +10.0 pp 6/10 E Β· 1/9 M Β· 1/1 H 1013.6 β†’ 1075.3 6.5

Fair-evaluation parity note: qwen35_0_8b stops cleanly on [248046 (<|im_end|>), 248044 (<|endoftext|>)] (eliminating multi-turn <tool_response> hallucination loops where pre-fix Base emitted 2068.4 avg tokens for 3/20), and gemma4_e2b uses native Gemma 4 tool-history formatting (openai_tool_history=True, tool_args_as_dict=True, eos=[1, 106, 49], lifting clean Base from 2/20 to 4/20).