Instructions to use MaziyarPanahi/Qwen3.5-4B-Judge-1M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use MaziyarPanahi/Qwen3.5-4B-Judge-1M with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("unsloth/Qwen3.5-4B") model = PeftModel.from_pretrained(base_model, "MaziyarPanahi/Qwen3.5-4B-Judge-1M") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Unsloth Desktop
Qwen3.5-4B-Judge-1M
A 4B open judge for everyday LLM checks: is this answer supported by its source?, is this request or response unsafe?, and which task or domain is this? It returns a calibrated probability for each option, not free text.
Qwen3.5-4B + an Unsloth decision head (LoRA r=16), trained on 39,592 rows of OpenDecisionMix-1M with source labels only. No Jev answers were used in training.
Results
The Oct 21 test file, 6,035 rows, scored per row with calibrated answers, gold = the human / source label. Every row that shared a state or a group with this file was removed from training first (5,645 rows).
| Test | Rows | This model | Qwen3.5-4B-Judge (Oct 21, human labels) | Jev 1.14 |
|---|---|---|---|---|
| RAG claim checks | 1,873 | 89.1% | 87.9% | 82.1% |
| Safety decisions | 2,162 | 89.8% | 90.7% | 82.3% |
| RAG + safety | 4,035 | 89.5% | 89.4% | 82.2% |
| MedHallu PubMed answers, at the 0.5 cut | 2,000 | 76.8% | 78.4% | 67.9% |
| MedHallu, real answer ranked above the fake one | 1,000 pairs | 862 | 855 | 916 |
- On RAG + safety it matches the Oct 21 judge (3,611 vs 3,609 rows right) while being trained on a broad mix of 20 task families instead of those two sets alone.
- Jev 1.14 ranks real PubMed answers above fake ones better (916 vs 862). Jev's default cut is strict on PubMed, so don't read 76.8% vs 67.9% on its own. Tune the threshold for your data.
- Jev 1.14's answers are the ones from the Oct 21 run (TypeSafe, no fallbacks), same state, instructions and criteria.
On the 1M's own test split (1,927 rows, 15 families), the families with enough rows:
| Family | Rows | Accuracy |
|---|---|---|
| Intent routing (banking) | 300 | 90.7% |
| Conversation domain routing | 278 | 91.7% |
| Pairwise response preference | 169 | 69.8% |
| Principle fulfilment | 125 | 78.4% |
| When to call a tool | 300 | 50.0% |
when_to_call_tool was not in its training sample and sits at chance: don't use it for that. Families with 12 test rows
or 100% on what may be a single class say nothing and are left out.
Use
from unsloth import FastDecisionModel
model, tokenizer = FastDecisionModel.from_pretrained("MaziyarPanahi/Qwen3.5-4B-Judge-1M", max_seq_length=4096, load_in_4bit=False)
FastDecisionModel.for_inference(model)
question = {"decision": {"type": "choice",
"instructions": "Is every factual claim in `assistant_response` supported by `authoritative_context`?",
"criteria": {"true": "Every factual claim in the response is supported by the context.",
"false": "The response contains at least one unsupported or contradicted claim."}}}
state = '{"authoritative_context": "The clinic opens at 8 a.m. on weekdays and is closed on Sundays.", '\
'"assistant_response": "The clinic is open every day from 8 a.m."}'
print(FastDecisionModel.predict(model, tokenizer, state, question))
Training
- Data: 39,992 rows sampled from the
choiceandnouldecisions of OpenDecisionMix-1M, capped at 2,505 per task family, states up to 5,000 characters, seed 3407. 39,592 train, 400 held out to calibrate. Labels are the source labels; everyjev_*column was removed from the training copy. - Recipe: Unsloth
FastDecisionModel, bf16 LoRA r=16, lr 2e-4, batch 8 x 4, max length 4,096. - Hardware: one A100 on Hugging Face Jobs, 1 h 40 min of training, stopped by a time budget at 0.97 of the epoch (about $5.30 at the listed $2.50/h).
Limits
- Not a clinical model. PubMed answers are a transfer test, not medical validation. It must not decide anything about a patient.
- Home advantage. The RAG and safety test rows come from the same sources as its training rows. Jev 1.14 answers zero-shot. The Oct 21 judge has the same advantage, so that column is the fair comparison.
- Long inputs are untested: training states were at most 5,000 characters, and the model reads at most 4,096 tokens.
Credit
- Base model: Qwen3.5-4B by the Qwen team, Apache-2.0. Recipe: Unsloth
FastDecisionModel. - Training rows come from OpenDecisionMix-1M, built from: NVIDIA's HelpSteer3, Aegis AI Content Safety 2.0, Open-SWE-Traces and When2Call (all CC BY 4.0); PolyAI/banking77 (CC BY 4.0); and RAGTruth via wandb/RAGTruth-processed (recorded as MIT with upstream review required; its Hub card has no license tag). Each was turned into decision questions with one label each. See those datasets for their terms.
- MedHallu (UTAustin-AIHealth/MedHallu) was used for evaluation only.
- Downloads last month
- 35