# Architecture

CrossbowReviewer is [Qwen3.5-9B-Base](https://huggingface.co/Qwen/Qwen3.5-9B-Base) with a LoRA merged into the
weights. The architecture is unchanged: no added layers and no classification head. What differs is how the model is
trained and read out: **slot decoding**.

![Architecture](assets/architecture.png)

## Base model

| | |
|---|---|
| Type | causal language model (`Qwen3_5ForCausalLM`, text part only) |
| Parameters | 9B dense (LoRA is merged, so this file set has no adapter) |
| Layers / hidden size | 32 / 4096, FFN 12288 |
| Layout | 8 × (3 × Gated DeltaNet → 1 × Gated Attention) |
| Gated DeltaNet | 32 value heads, 16 QK heads, head dim 128, conv kernel 4 |
| Gated Attention | 16 query heads, 4 KV heads (GQA), head dim 256, partial RoPE (0.25) |
| Vocabulary | 248 320 |
| Context | 262 144 tokens natively |
| MTP head | **not shipped.** `config.json` still carries the base value `mtp_num_hidden_layers: 1`, but no MTP weights are in the repo and decisions never generate text |

Three of every four layers are linear attention, so prefill cost grows gently with file length. This is what makes a
single pass over a whole file and ~107 rules cheap.

## Slot decoding

1. **Prompt.** Header, then `state` (a JSON object with the language and the code), then every question as
   `### name`, instructions, lettered options and `Answer:`.
2. **One forward pass.** The answers are never in the input, so a slot cannot see the answers to earlier slots.
   Training and inference see the same sequence, and all questions share one prefill.
3. **Read-out.** The LM head runs only at the positions of the `Answer:` slots, not over the whole sequence. For each
   slot the logits of the option letters (` A`, ` B`, up to 26) are taken.
4. **Calibration.** `softmax(logits / T)` with a single temperature `T = 1.122` (`decision_config.json`). The server
   can add a per-category scale and accept/defer thresholds. For data-flow security no threshold could be certified.
5. **Typed output.** `noul`: yes/no (`no` means the rule is violated); `choice`: up to 26 options; `score`: 2–10
   levels. Every answer has a `choice`, a `confidence` and the full `distribution`.

## Training

- **Method:** LoRA on all linear modules (`q/k/v/o`, `gate/up/down` and the DeltaNet projections `in_proj_qkv`,
  `in_proj_z`, `in_proj_a`, `in_proj_b`, `out_proj`). Rank 32, alpha 64, dropout 0.05.
- **Loss:** cross-entropy over the option-letter logits at each slot. Each sample shows a random subset of the questions,
  always including its target rule.
- **Data:** 4 866 samples: synthetic snippets, 445 vulnerable/fixed pairs with a verified contrast and 900 real-code files
  with soft labels. 3 rounds × 1 epoch, lr 5e-5, bf16, 1× A100.

## What is not new

Unlike recent architecture releases, such as sparse attention, gated residuals or n-gram embeddings in
[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), CrossbowReviewer does not change the network. The
gain comes from data, the slot read-out and calibration, so any improvement in the base model can be reused by
retraining the same recipe.

## Variants we tried

All documented in the
[research log](https://github.com/riposta/CrossbowReviewer-Pipeline/blob/main/docs/RESEARCH_UPGRADE.md): a prefix layout
with a fixed rule list before the code (worse on OWASP injection and path traversal), contrast-format training (worse
on real code), averaging both option orders (no gain) and LoRA on basal-omni (collapsed to chance).