# Benchmarks (CrossbowReviewer 1.1)

Version 1.1 replaces the 1.0 benchmark with a stricter protocol. All numbers are also in
[benchmarks.json](benchmarks.json). The 1.0 numbers (macro-averaged per rule) are not comparable with the pooled
numbers below; the 1.0 card is available under the `v1.0` tag of this repository.

## What changed in how we test

1. **Pooled metrics.** Real-code labels are sparse (6.9% violations, 55 files from 9 repositories). A per-rule macro
   MCC over a handful of positives is noisy, so we pool all yes/no decisions of a test set and report MCC, AUC, average
   precision (AP), recall, specificity, balanced accuracy and ECE. `positive` = "the rule is violated".
2. **Cluster bootstrap.** Confidence intervals resample whole samples (or repositories), not single decisions. This
   keeps correlated questions of one file together.
3. **Paired differences.** 1.1 vs 1.0 is tested on the same decisions with a shared resample, so a gain is only
   reported when the interval excludes zero.
4. **A held-out OWASP set.** The 435 OWASP cases were used for model selection and calibration while we built 1.1. We
   drew 429 *new* OWASP Benchmark cases (`seed 1`, never used for training, selection or calibration) as an
   independent confirmation.
5. **Vulnerability/fix pairs.** 98 pairs from GitHub Security Advisories (vulnerable file and its fix). The model must
   flag the vulnerable version and accept the fixed one. This is the hardest test and the one the 1.0 model failed.
6. **Same files, same protocol** for every system. Predictions are cached and keyed by input; every system answers
   exactly the questions each sample is labeled for.

## Results

Pooled MCC with 95% cluster-bootstrap CIs:

![Pooled MCC](assets/release_mcc.png)

**Synthetic** (2 838 decisions, 691 violations)

| System | MCC (95% CI) | AUC | AP | Recall | Specificity | bal_acc | ECE |
|---|---|---:|---:|---:|---:|---:|---:|
| **CrossbowReviewer 1.1** | 0.72 (0.69–0.75) | 0.955 | 0.892 | 0.69 | 0.97 | 0.83 | 0.010 |
| CrossbowReviewer 1.0 | 0.72 (0.68–0.75) | 0.959 | 0.905 | 0.65 | 0.98 | 0.82 | 0.009 |
| Qwen3.5-9B-Base (zero-shot) | 0.26 (0.21–0.30) | 0.653 | 0.463 | 0.36 | 0.87 | 0.62 | 0.040 |
| Jev 1.14.0 (latest) | 0.58 (0.55–0.62) | 0.891 | 0.706 | 0.81 | 0.83 | 0.82 | 0.054 |
| Jev 1.13.0 | 0.57 (0.54–0.61) | 0.883 | 0.684 | 0.81 | 0.82 | 0.81 | 0.046 |
| basal-omni-1.0-max (zero-shot) | 0.60 (0.56–0.64) | 0.896 | 0.771 | 0.71 | 0.89 | 0.80 | 0.076 |
| Claude Opus 5.5 | 0.85 (0.82–0.87) | 0.985 | 0.960 | 0.86 | 0.97 | 0.92 | 0.120 |
| GPT-6 Sol | 0.80 (0.78–0.83) | 0.976 | 0.930 | 0.95 | 0.91 | 0.93 | 0.021 |

**Real open-source code** (5 361 decisions, 370 violations; judge labels from frontier LLMs, so Opus/GPT scores are
likely inflated)

| System | MCC (95% CI) | AUC | AP | Recall | Specificity | bal_acc | ECE |
|---|---|---:|---:|---:|---:|---:|---:|
| **CrossbowReviewer 1.1** | 0.51 (0.46–0.54) | 0.961 | 0.705 | 0.36 | 0.99 | 0.67 | 0.014 |
| CrossbowReviewer 1.0 | 0.47 (0.42–0.53) | 0.938 | 0.576 | 0.35 | 0.99 | 0.67 | 0.006 |
| Qwen3.5-9B-Base (zero-shot) | -0.13 (-0.16–-0.10) | 0.309 | 0.046 | 0.04 | 0.75 | 0.39 | 0.029 |
| Jev 1.14.0 (latest) | 0.25 (0.21–0.28) | 0.801 | 0.178 | 0.68 | 0.75 | 0.72 | 0.031 |
| Jev 1.13.0 | 0.25 (0.21–0.28) | 0.799 | 0.174 | 0.67 | 0.76 | 0.72 | 0.030 |
| basal-omni-1.0-max (zero-shot) | 0.26 (0.22–0.31) | 0.802 | 0.224 | 0.54 | 0.85 | 0.70 | 0.066 |
| Claude Opus 5.5 | 0.67 (0.63–0.72) | 0.990 | 0.905 | 0.49 | 1.00 | 0.74 | 0.086 |
| GPT-6 Sol | 0.84 (0.81–0.87) | 0.981 | 0.886 | 0.83 | 0.99 | 0.91 | 0.037 |

**OWASP Benchmark, 435 cases** (used for model selection in 1.1)

| System | MCC (95% CI) | AUC | AP | Vulnerable flagged | Safe recognized | bal_acc | ECE |
|---|---|---:|---:|---:|---:|---:|---:|
| **CrossbowReviewer 1.1** | 0.30 (0.22–0.40) | 0.697 | 0.715 | 0.74 | 0.55 | 0.65 | 0.168 |
| CrossbowReviewer 1.0 | 0.28 (0.20–0.36) | 0.715 | 0.722 | 0.94 | 0.26 | 0.60 | 0.239 |
| Qwen3.5-9B-Base (zero-shot) | 0.37 (0.29–0.45) | 0.727 | 0.713 | 0.93 | 0.37 | 0.65 | 0.025 |
| Jev 1.14.0 (latest) | 0.33 (0.25–0.40) | 0.725 | 0.687 | 0.95 | 0.30 | 0.62 | 0.178 |
| Jev 1.13.0 | 0.30 (0.22–0.37) | 0.719 | 0.649 | 0.97 | 0.24 | 0.60 | 0.209 |
| basal-omni-1.0-max (zero-shot) | 0.50 (0.45–0.56) | 0.825 | 0.800 | 1.00 | 0.40 | 0.70 | 0.257 |
| Claude Opus 5.5 | 0.94 (0.90–0.97) | 0.983 | 0.977 | 1.00 | 0.94 | 0.97 | 0.059 |
| GPT-6 Sol | 0.77 (0.72–0.82) | 0.935 | 0.905 | 1.00 | 0.75 | 0.88 | 0.106 |

**OWASP Benchmark, 429 held-out cases** (independent confirmation)

| System | MCC (95% CI) | AUC | AP | Vulnerable flagged | Safe recognized | bal_acc | ECE |
|---|---|---:|---:|---:|---:|---:|---:|
| **CrossbowReviewer 1.1** | 0.34 (0.23–0.42) | 0.722 | 0.700 | 0.88 | 0.42 | 0.65 | 0.150 |
| CrossbowReviewer 1.0 | 0.34 (0.25–0.41) | 0.738 | 0.750 | 0.94 | 0.32 | 0.63 | 0.238 |
| Qwen3.5-9B-Base (zero-shot) | 0.39 (0.31–0.45) | 0.744 | 0.733 | 0.93 | 0.41 | 0.67 | 0.027 |
| Jev 1.14.0 (latest) | 0.34 (0.25–0.41) | 0.763 | 0.719 | 0.94 | 0.32 | 0.63 | 0.189 |

**Vulnerability/fix pairs** (98 pairs = 196 decisions)

| System | MCC (95% CI) | AUC | Vulnerable flagged | Fixed accepted | Pair accuracy | ECE |
|---|---|---:|---:|---:|---:|---:|
| **CrossbowReviewer 1.1** | 0.24 (0.14–0.34) | 0.632 | 0.39 | 0.83 | **0.235** | 0.197 |
| CrossbowReviewer 1.0 | 0.08 (-0.03–0.19) | 0.506 | 0.05 | 0.98 | 0.041 | 0.353 |
| Qwen3.5-9B-Base (zero-shot) | 0.05 (0.00–0.14) | 0.529 | 0.05 | 0.97 | – | 0.247 |
| Jev 1.14.0 (latest) | 0.10 (0.03–0.17) | 0.546 | 0.50 | 0.60 | 0.112 | 0.220 |
| basal-omni-1.0-max (zero-shot) | 0.14 (0.05–0.21) | 0.574 | 0.47 | 0.66 | 0.163 | 0.319 |

Pair accuracy = both versions of a pair judged correctly.

## 1.1 vs 1.0: paired differences

| Test | ΔMCC (95% CI) | ΔAUC (95% CI) | Δbal_acc (95% CI) | Verdict |
|---|---|---|---|---|
| Synthetic | +0.005 (−0.020, +0.029) | −0.004 (−0.009, +0.001) | +0.012 (−0.002, +0.026) | no change |
| Real code | +0.036 (−0.023, +0.099) | **+0.023 (+0.014, +0.032)** | +0.003 (−0.012, +0.020) | better ranking |
| OWASP (435) | +0.021 (−0.079, +0.123) | −0.018 (−0.054, +0.023) | +0.045 (0.000, +0.091) | borderline |
| OWASP held-out | −0.001 (−0.083, +0.069) | −0.016 (−0.063, +0.024) | +0.018 (−0.017, +0.052) | no change |
| Vuln/fix pairs | **+0.156 (+0.036, +0.301)** | **+0.126 (+0.074, +0.194)** | **+0.092 (+0.046, +0.148)** | clearly better |

## What the benchmarks say

- **Confirmed gains:** vulnerability/fix pairs (pair accuracy 0.04 → 0.235), ranking of real violations (AUC
  0.938 → 0.961, AP 0.58 → 0.71) and calibration on security data (ECE 0.24 → 0.15–0.17).
- **No gain:** synthetic data and OWASP. On the held-out OWASP set 1.1 equals 1.0 (MCC 0.34 both). The small OWASP
  improvement seen on the 435 selection set did not replicate, so we do not claim it.
- **The zero-shot base model is as good as both fine-tunes on OWASP** (MCC 0.39 held-out, within the CIs). Fine-tuning
  does not teach single-pass data-flow reasoning. basal-omni zero-shot is clearly better on OWASP (MCC 0.50 on the 435
  set; not run on the held-out set), and the frontier reasoning models are far ahead (Opus 5.5: 0.94).
- **Security stays a human/reasoning-model decision.** 1.1 recognizes 42–55% of the safe OWASP traps (1.0: 26–32%),
  better but still far from Opus 5.5 (94%).

## Against the latest Jev

We re-ran everything with **jev-1.14.0** (the current `jev-latest`); 1.0's card used jev-1.13.0, which is kept in the
tables for reference. 1.14.0 is only marginally better than 1.13.0 on every set. CrossbowReviewer 1.1 stays well ahead
of it on real code (MCC 0.51 vs 0.25, AUC 0.961 vs 0.801, AP 0.71 vs 0.18) and on vulnerability/fix pairs (pair
accuracy 0.235 vs 0.112) and on synthetic data (MCC 0.72 vs 0.58). Jev flags more violations (recall 0.81 vs 0.69
on synthetic, 0.68 vs 0.36 on real code) but pays with 25% false positives on real code versus 1%. On OWASP the two
are statistically indistinguishable (held-out MCC 0.34 vs 0.34).

## Other benchmarks considered

We surveyed public benchmarks that could extend this evaluation. None is included in the numbers above.

| Benchmark | Why it is useful | Status |
|---|---|---|
| PrimeVul paired test (C/C++) | de-duplicated vulnerable/fixed pairs; paired metrics P-C, P-V, P-B, P-R | planned; out of language distribution |
| OWASP BenchmarkJava, remaining ~1 900 cases | larger Java security set | partly used (435 + 429 held-out) |
| Juliet Java 1.3 good/bad pairs | CWE coverage | templated code, sanity check only |
| CodeReviewer-style comment generation | not applicable: our model returns typed decisions, not comments | not used |
| CVEfixes, MegaVul, DiverseVul, Big-Vul | large | not used: label noise and training-data contamination risk |

## Latency and size

| System | Seconds per file | Hardware |
|---|---:|---|
| CrossbowReviewer (transformers, BF16) | ~0.2 | 1× A100 40 GB |
| CrossbowReviewer (llama.cpp, Q8_0) | ~3 | Apple Silicon (Metal) |
| Claude Opus 5.5 | 8.2 | API |
| GPT-6 Sol | 9.8 | API |

## Reproducibility

- **Weights:** the merged BF16 safetensors match base + LoRA recorded on a GPU (50 samples, 1030 decisions): 1026/1030 identical choices
  (mean |Δp| 0.002, max 0.04). The Q8_0 GGUF matches the 1.1 predictions on 40/40 decisions (mean |Δp| 0.008).
- **Test data is not redistributed:** OWASP Benchmark is GPL-2.0 and the source files belong to their authors.
  The evaluation code (`evalkit.py`, `release_bench.py`, `owasp.py --exclude`) is in the
  [pipeline repository](https://github.com/riposta/CrossbowReviewer-Pipeline).
- **Limits of the evidence:** real code is 55 files from 9 repositories (7 clusters for per-repo bootstrap), 98 pairs,
  one seed per model. Differences below ~0.05 MCC are not distinguishable.