docs/ARCHITECTURE.md

Architecture

CrossbowReviewer is Qwen3.5-9B-Base with a LoRA merged into the weights. The architecture is unchanged: no added layers and no classification head. What differs is how the model is trained and read out: slot decoding.

Base model

Type	causal language model (Qwen3_5ForCausalLM, text part only)
Parameters	9B dense (LoRA is merged, so this file set has no adapter)
Layers / hidden size	32 / 4096, FFN 12288
Layout	8 × (3 × Gated DeltaNet → 1 × Gated Attention)
Gated DeltaNet	32 value heads, 16 QK heads, head dim 128, conv kernel 4
Gated Attention	16 query heads, 4 KV heads (GQA), head dim 256, partial RoPE (0.25)
Vocabulary	248 320
Context	262 144 tokens natively
MTP head	not shipped. config.json still carries the base value mtp_num_hidden_layers: 1, but no MTP weights are in the repo and decisions never generate text

Three of every four layers are linear attention, so prefill cost grows gently with file length. This is what makes a single pass over a whole file and ~107 rules cheap.

Slot decoding

1. Prompt. Header, then state (a JSON object with the language and the code), then every question as ### name, instructions, lettered options and Answer:.
2. One forward pass. The answers are never in the input, so a slot cannot see the answers to earlier slots. Training and inference see the same sequence, and all questions share one prefill.
3. Read-out. The LM head runs only at the positions of the Answer: slots, not over the whole sequence. For each slot the logits of the option letters ( A, B, up to 26) are taken.
4. Calibration. softmax(logits / T) with a single temperature T = 1.122 (decision_config.json). The server can add a per-category scale and accept/defer thresholds. For data-flow security no threshold could be certified.
5. Typed output. noul: yes/no (no means the rule is violated); choice: up to 26 options; score: 2–10 levels. Every answer has a choice, a confidence and the full distribution.

Training

• Method: LoRA on all linear modules (q/k/v/o, gate/up/down and the DeltaNet projections in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj). Rank 32, alpha 64, dropout 0.05.
• Loss: cross-entropy over the option-letter logits at each slot. Each sample shows a random subset of the questions, always including its target rule.
• Data: 4 866 samples: synthetic snippets, 445 vulnerable/fixed pairs with a verified contrast and 900 real-code files with soft labels. 3 rounds × 1 epoch, lr 5e-5, bf16, 1× A100.

What is not new

Unlike recent architecture releases, such as sparse attention, gated residuals or n-gram embeddings in Qwen3.8-Flash-Next, CrossbowReviewer does not change the network. The gain comes from data, the slot read-out and calibration, so any improvement in the base model can be reused by retraining the same recipe.

Variants we tried

All documented in the research log: a prefix layout with a fixed rule list before the code (worse on OWASP injection and path traversal), contrast-format training (worse on real code), averaging both option orders (no gain) and LoRA on basal-omni (collapsed to chance).