Qwen3.5-4B-Judge-1M

A 4B open judge for everyday LLM checks: is this answer supported by its source?, is this request or response unsafe?, and which task or domain is this? It returns a calibrated probability for each option, not free text.

Qwen3.5-4B + an Unsloth decision head (LoRA r=16), trained on 39,592 rows of OpenDecisionMix-1M with source labels only. No Jev answers were used in training.

Results

The Oct 21 test file, 6,035 rows, scored per row with calibrated answers, gold = the human / source label. Every row that shared a state or a group with this file was removed from training first (5,645 rows).

Test Rows This model Qwen3.5-4B-Judge (Oct 21, human labels) Jev 1.14
RAG claim checks 1,873 89.1% 87.9% 82.1%
Safety decisions 2,162 89.8% 90.7% 82.3%
RAG + safety 4,035 89.5% 89.4% 82.2%
MedHallu PubMed answers, at the 0.5 cut 2,000 76.8% 78.4% 67.9%
MedHallu, real answer ranked above the fake one 1,000 pairs 862 855 916
  • On RAG + safety it matches the Oct 21 judge (3,611 vs 3,609 rows right) while being trained on a broad mix of 20 task families instead of those two sets alone.
  • Jev 1.14 ranks real PubMed answers above fake ones better (916 vs 862). Jev's default cut is strict on PubMed, so don't read 76.8% vs 67.9% on its own. Tune the threshold for your data.
  • Jev 1.14's answers are the ones from the Oct 21 run (TypeSafe, no fallbacks), same state, instructions and criteria.

On the 1M's own test split (1,927 rows, 15 families), the families with enough rows:

Family Rows Accuracy
Intent routing (banking) 300 90.7%
Conversation domain routing 278 91.7%
Pairwise response preference 169 69.8%
Principle fulfilment 125 78.4%
When to call a tool 300 50.0%

when_to_call_tool was not in its training sample and sits at chance: don't use it for that. Families with 12 test rows or 100% on what may be a single class say nothing and are left out.

Use

from unsloth import FastDecisionModel

model, tokenizer = FastDecisionModel.from_pretrained("MaziyarPanahi/Qwen3.5-4B-Judge-1M", max_seq_length=4096, load_in_4bit=False)
FastDecisionModel.for_inference(model)
question = {"decision": {"type": "choice",
    "instructions": "Is every factual claim in `assistant_response` supported by `authoritative_context`?",
    "criteria": {"true": "Every factual claim in the response is supported by the context.",
                 "false": "The response contains at least one unsupported or contradicted claim."}}}
state = '{"authoritative_context": "The clinic opens at 8 a.m. on weekdays and is closed on Sundays.", '\
        '"assistant_response": "The clinic is open every day from 8 a.m."}'
print(FastDecisionModel.predict(model, tokenizer, state, question))

Training

  • Data: 39,992 rows sampled from the choice and noul decisions of OpenDecisionMix-1M, capped at 2,505 per task family, states up to 5,000 characters, seed 3407. 39,592 train, 400 held out to calibrate. Labels are the source labels; every jev_* column was removed from the training copy.
  • Recipe: Unsloth FastDecisionModel, bf16 LoRA r=16, lr 2e-4, batch 8 x 4, max length 4,096.
  • Hardware: one A100 on Hugging Face Jobs, 1 h 40 min of training, stopped by a time budget at 0.97 of the epoch (about $5.30 at the listed $2.50/h).

Limits

  • Not a clinical model. PubMed answers are a transfer test, not medical validation. It must not decide anything about a patient.
  • Home advantage. The RAG and safety test rows come from the same sources as its training rows. Jev 1.14 answers zero-shot. The Oct 21 judge has the same advantage, so that column is the fair comparison.
  • Long inputs are untested: training states were at most 5,000 characters, and the model reads at most 4,096 tokens.

Credit

Downloads last month
35
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for MaziyarPanahi/Qwen3.5-4B-Judge-1M

Finetuned
Qwen/Qwen3.5-4B
Adapter
(245)
this model

Dataset used to train MaziyarPanahi/Qwen3.5-4B-Judge-1M