CrossbowReviewer-9B 1.1

Calibrated code review decisions in one forward pass. It asks up to 107 rules per file, runs in ~0.2 s on an A100 and in ~3 s on a Mac with the Q8_0 GGUF.

CrossbowReviewer-9B fine-tunes Qwen3.5-9B-Base for Jev-style typed decisions about code. It writes no review text. For each question, such as "is this free of SQL injection?" or "does each class have one reason to change?", it returns a choice, a calibrated confidence and the full distribution. Its inference server provides a /v1/systemone endpoint that follows the Jev request schema.

Inference server · Quickstart · Architecture · Benchmarks · Artifact manifest · 1.0 is kept under the v1.0 tag.

What's new in 1.1

Same architecture, format and API as 1.0 (drop-in replacement). The training data and recipe changed.

	1.0	1.1
Training data	1 466 synthetic snippets	4 866 samples: synthetic + 445 vulnerable/fixed pairs with a verified contrast + 900 real-code files with soft labels
LoRA	rank 16, alpha 32, lr 1e-4	rank 32, alpha 64, lr 5e-5
Temperature	1.044	1.122
Evaluation	macro per-rule metrics	pooled metrics, cluster bootstrap, paired tests, a held-out OWASP set, vuln/fix pairs

Measured improvements (1.1 vs 1.0, paired, 95% CI):

Test	Result
Vulnerability/fix pairs (98)	pair accuracy 0.04 → 0.235; vulnerable flagged 0.05 → 0.39; ΔMCC +0.16 (+0.04, +0.30)
Real code	AUC 0.938 → 0.961 (Δ +0.023, CI +0.014…+0.032); AP 0.58 → 0.71; MCC 0.47 → 0.51 (CI includes 0)
OWASP safe-recognized (taint traps)	0.26 → 0.55 on the 435 set, 0.32 → 0.42 on the held-out set
Calibration on security data	ECE 0.24 → 0.15–0.17
Synthetic	unchanged (MCC 0.72 both)

Not improved, said plainly: on an independent held-out OWASP set (429 cases) 1.1 and 1.0 are equal (MCC 0.34 both), and the untuned Qwen3.5-9B-Base is as good (0.39). Data-flow security is not something this model learns in one pass. See docs/BENCHMARKS.md for all tables.

Features

• Typed decisions: noul (yes/no), choice (up to 26 options) and score (2–10 levels), asked against the code as state.
• One pass: all questions share one prefill. Option-letter logits are read at each answer slot, so no answer text is generated.
• 120 built-in rules: SOLID and design principles, patterns and anti-patterns, correctness, error handling, security (OWASP / CWE), performance, readability, clean code, testability, testing, API design and language idioms. Two aggregate questions are included (primary_category, severity).
• 7 languages: Java, Python, TypeScript, JavaScript, Go, C#, Rust.
• Full weights: no adapter and no conversion step. Safetensors work with transformers; GGUF works with llama.cpp.

Architecture

Qwen3.5-9B-Base (32 layers, 8 × (3 Gated DeltaNet + 1 Gated Attention), hidden 4096) with a rank-32 LoRA merged into the weights. No added layers or heads: all questions go in one prompt, the logits of the option letters are read at every Answer: slot in a single forward pass, and a fitted temperature turns them into calibrated probabilities.

Details, base-model table and training recipe: docs/ARCHITECTURE.md.

Downloads

Files	Format and use	Size
model-0000{1..4}-of-00004.safetensors + configs	BF16 safetensors. transformers backend (CUDA, Apple MPS, CPU)	17.9 GB
gguf/CrossbowReviewer-9B-Q8_0.gguf	Q8_0 GGUF. llama.cpp backend; fits 12 GB GPUs and Apple Silicon	9.5 GB

decision_config.json holds what turns logits into decisions: the fitted temperature (1.122), the option letters and the prompt format. Both backends read it. If you cache 1.0's decision_config.json, refresh it.

Parity: the merged safetensors match base + LoRA on 1026/1030 decisions (mean |Δp| 0.002); the Q8_0 GGUF matches the safetensors predictions on 40/40 decisions (mean |Δp| 0.008).

Quick start

git clone https://github.com/riposta/CrossbowReviewer && cd CrossbowReviewer
uv run uvicorn server:app --port 8000     # transformers backend; downloads this repo on first request
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \
  -d '{"state": {"language": "python", "code": "def add(items=[]):\n    items.append(1)\n    return items"}}'

Without questions, every rule for state.language is asked. yes means compliant:

{"decisions": {"py_mutable_default": {"choice": "no", "confidence": 0.98, "distribution": {"yes": 0.02, "no": 0.98}}, "...": "..."}}

To pin 1.0, use revision="v1.0" in from_pretrained or hf download --revision v1.0. For llama.cpp, custom questions and all options, see docs/QUICKSTART.md.

Comparison

Pooled MCC (1 = perfect, 0 = chance), 95% cluster-bootstrap CI in docs/BENCHMARKS.md:

System	Synthetic	Real code	OWASP held-out	Vuln/fix pairs
CrossbowReviewer 1.1	0.72	0.51	0.34	0.24
CrossbowReviewer 1.0	0.72	0.47	0.34	0.08
Qwen3.5-9B-Base (zero-shot)	0.26	−0.13	0.39	0.05
Jev 1.14.0 (latest)	0.58	0.25	0.34	0.10
Jev 1.13.0	0.57	0.25	–	–
basal-omni-1.0-max (zero-shot)	0.60	0.26	0.50¹	0.14
Claude Opus 5.5 (reasoning)	0.85	0.67	0.94¹	–
GPT-6 Sol (reasoning)	0.80	0.84²	0.77¹	–

¹ OWASP 435-case set (held-out set not run for these systems). ² Real-code labels come from frontier judges of the same families, so frontier scores there are likely inflated.

• Calibration: ECE is 0.010 on synthetic and 0.014 on real code. When the model says 0.9, it is right about 90% of the time.
• Latency: reasoning models are better on security, but 25–50× slower (8–10 s per file versus ~0.2 s).

Limitations — read before use

Security taint traps. On OWASP the model flags 74–88% of vulnerable cases and recognizes only 42–55% of the safe cases built to look vulnerable (tainted input reaching only a dead branch, overwritten by a constant, and so on). 1.0 recognized 26–32%. One forward pass cannot trace data flow the way a reasoning model can (Opus 5.5: 94%), and the zero-shot base model is no worse than the fine-tune on this task. Treat no on data-flow security rules (injection, path_traversal, xss, ssrf, open_redirect) as "needs a closer look" and escalate to a reasoning model or a human. On real-world-like security code, no confidence threshold was certifiable, so security decisions should always be reviewed.

Recommended use. Use the model directly for design, readability, correctness, testing and idiom rules.

Other limits:

• Mostly synthetic and LLM-labeled data. Real-code labels are soft labels from LLM judges, not human reviews of real pull requests. Synthetic scores overstate real-world accuracy.
• Recall on real violations is 0.36 at the default threshold (specificity 0.99). The model is conservative on real code; use the distribution (AUC 0.96) to tune your threshold.
• Small evaluation sets. Real code is 55 files from 9 repositories, pairs are 98, one seed per model. Differences below ~0.05 MCC are not distinguishable.
• One file at a time. The model judges snippets, not diffs, and does not see the rest of the codebase.
• Custom questions outside the trained rules use the same format, but their quality is not measured.
• Prompts are in English. choice supports up to 26 options (Jev: 255).

What did not work (so you don't repeat it)

Documented in the research log: more security-only pairs (v4: better recall, worse calibration), contrast-format training with a fixed rule prefix (v6: worse on real code and OWASP), moving the rule list before the code (prefix layout: worse on OWASP injection and path traversal), averaging both option orders, and LoRA on basal-omni (collapsed to chance).

Training

• Data: 4 866 samples: synthetic snippets (language × rule × compliant/violating, labeled by an LLM), 445 vulnerable/fixed pairs from GitHub Security Advisories whose contrast was confirmed by two frontier judges, and 900 real-code files with soft labels.
• Method: LoRA on all linear layers (rank 32, alpha 64), merged into these full weights. Cross-entropy over option-letter logits at every answer slot. Each sample shows a random subset of questions, always including its target rule.
• Optimization: 3 rounds × 1 epoch, AdamW, lr 5e-5, bf16, 1× A100 (GCP). The best round was selected on a validation split.
• Calibration: a single temperature (1.122) fitted on a validation split.
• Not included: the multi-token-prediction head of the base model. No MTP weights are in this repo (config.json still carries the base value mtp_num_hidden_layers: 1), because decisions never generate text.

License

Apache 2.0, the same as the base model. See LICENSE and NOTICE.