OpenDecider: Open, Calibrated Decision Models That Beat Laya and Jev on Typed-Decisions
Decision models ("System One" models) answer bounded questions about an input and return a probability for every allowed answer: which team should handle this ticket, how urgent is it, is this email phishing. TypeSafe's Jev made the interface popular as a closed API; Laya followed as an open model. This article introduces OpenDecider, a family of open decision models, and reports how it compares with Jev, Laya, Stanford and NVIDIA's CLM-8B and frontier LLMs when all of them answer the same questions and are scored by the same code.
The short version. OpenDecider-nano (~400M) scores 0.796 on typed-decisions, against 0.766 for Laya's fine-tuned checkpoint (paired bootstrap 95% CI for the gap: +0.014 to +0.044) and 0.754 for Jev measured through TypeSafe's own API. OpenDecider-small (4B) matches Jev on 200 general decisions none of these models trained on (0.735 vs 0.730) and is the best-calibrated self-hostable model I measured. It is not better everywhere: Jev leads Laya's own application battery, and phishing is OpenDecider's weakest task. Everything, including every logged answer from every system, is public.
Update, October 3, 2026 (0.6.0). Since this article was published:
- Two larger models. opendecider-medium-td (Qwen3-30B-A3B) scores 0.765 on the 200 general decisions, ahead of Jev's 0.730, the best of any model you can run yourself. opendecider-large-td (Qwen3-Next-80B-A3B) has the best calibration (ECE 0.083) and is the closest to human judgement (ChaosNLI JSD 0.030). Both need NVIDIA GPUs.
- Ollama and LM Studio: GGUF builds of small and small-td (Q8_0 and Q4_K_M).
- A prompt guard against jailbreaks and prompt injection, in each agent framework's own hook (0.5.0).
- TypeScript: npm i @opendecider/client, with tools for the Vercel AI SDK and Mastra (0.6.0). For Python apps without PyTorch: pip install opendecider-client.
- Self-hosting: opendecider serve, a drop-in for Jev's /v1/systemone API, as a signed Docker image for x86 and ARM.
The original measurements below are unchanged; the new models are added to the tables.
Try it
pip install opendecider # nano
pip install "opendecider[small]" # the 4B models
pip install "opendecider[mlx]" # Apple Silicon MLX builds
from opendecider import load
model = load("manjunathshiva/opendecider-nano")
result = model.system_one(
"Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan.",
{
"department": {"type": "choice", "instructions": "Which department should handle this?",
"criteria": {"billing": "invoices, payments, refunds",
"technical": "bugs, outages, system errors",
"other": "everything else"}},
"urgency": {"type": "score", "instructions": "How urgent is this?",
"criteria": ["not urgent", "soon", "blocking"]},
"churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"},
})
print(result["answers"]["department"]["choice"]) # billing (0.927)
print(result["answers"]["urgency"]["score"]) # 2 = blocking (0.604)
print(result["answers"]["churn_risk"]["noul"]) # 0.922
No install needed for a first look: the live demo runs nano on a CPU Space, and a Colab notebook runs both models on a free GPU (tested top to bottom on an NVIDIA GPU; small switches to fp16 on a T4).
The models
| model | size | memory | best for |
|---|---|---|---|
| opendecider-nano | ~400M (Ettin encoder) | 2.0 GiB | speed: 16 ms per question on an L40S, ~4 ms batched; business workflow decisions |
| opendecider-small | 4B (Qwen3-4B + LoRA) | 8.9 GiB, tested on a 16 GB Mac mini | decisions it has never seen: best accuracy and calibration |
| opendecider-small-td | 4B | 8.9 GiB | business workflows like typed-decisions': 0.792 |
| opendecider-small-mlx-8bit | 4B, MLX 8-bit | 4.5 GB | Macs: same top answer as full precision on 1,955 of 2,000 typed-decisions questions, 66 ms |
| opendecider-small-mlx-4bit | 4B, MLX 4-bit | 2.6 GB | Macs with little memory; about 2 points lower on typed-decisions |
| opendecider-medium-td | 30B (3B active), Qwen3-30B-A3B + LoRA | 61 GB bf16, NVIDIA (tested on 4× L40S) | general decisions: 0.765, the best self-hostable model; closest to human judgement |
| opendecider-large-td | 80B (3B active), Qwen3-Next-80B-A3B + LoRA | 160 GB bf16, NVIDIA (tested on 4× L40S) | thresholding on probabilities: best calibration |
| opendecider-small-GGUF | 4B, GGUF Q8_0 / Q4_K_M | 4.3 / 2.5 GB | LM Studio and Ollama; typed-decisions 0.669 at Q8_0 (full precision 0.671) |
| opendecider-small-td-GGUF | 4B, GGUF Q8_0 / Q4_K_M | 4.3 / 2.5 GB | LM Studio and Ollama, business workflows: 0.794 at Q8_0 (full precision 0.792) |
All are Apache-2.0 and give identical answers on Apple Silicon (MPS), NVIDIA (CUDA) and CPU.
How they work
OpenDecider-nano reads question: …, [MASK] option 1, [MASK] option 2, …, input: <state> in one encoder pass. The hidden state at each [MASK] goes through a small MLP to one logit, then a softmax across the question's options. This option-marker idea is the same one Laya uses; the difference is that there is no fixed token budget shared by the options, so a 78-option question still gets every option in full. That is why nano scores 0.645 on 77-label BANKING77 where Laya scores 0.425.
OpenDecider-small is Qwen3-4B-Instruct-2507 with a LoRA adapter. The options are lettered, and one forward pass reads the probability of each letter as the next token; above 26 options it scores each option name's log-likelihood.
Both are trained by distillation from calibrated teachers. Two openly licensed teachers, Qwen3-235B-A22B-Instruct-2507 (Apache-2.0) and DeepSeek V4.1 Flash (MIT), answered every training question with lettered options, and their option distributions were read from next-token log-probabilities. Each teacher was temperature-scaled on held-out gold labels before the two were averaged, so the students learn calibrated distributions rather than hard labels. The effect shows directly: untrained Qwen3-4B has an expected calibration error of 0.289; OpenDecider-small has 0.087.
The training data (~190K questions) combines public classification, NLI, reading-comprehension, topic, toxicity, spam, relevance and paraphrase datasets with synthetic business cases, emails and product reviews. Every dataset used for evaluation, and its family, is excluded, and every training pool was checked for text overlap with all test sets: 0 overlaps. nano and small-td then had a short fine-tune on the typed-decisions train split, like Laya's typed-decisions checkpoint; the test split was never used for training or model selection. Training ran on AWS SageMaker (NVIDIA L40S) for under $30 of GPU time.
How they compare
Every system answered the same questions and was scored by the same code:
- Jev through TypeSafe's own API with native typed questions, not published figures.
- typed-decisions scored with the Jev-vs-Laya harness published by Kameshwara Pavan kumar Mantha and the Antz AI team, joined question by question with their results (all 2,000 gold labels matched).
- Laya's application battery run through Laya's own
bench_apps.py, unchanged, at a pinned commit. - CLM-8B run with its own engine through the official vLLM pooling server.
| typed-decisions (2,000) | 200 general decisions | Laya battery (10 × 400) | calibration error ↓ | latency | |
|---|---|---|---|---|---|
| OpenDecider-nano | 0.796 | 0.680 | 0.656 | 0.092 | 16 ms (L40S) |
| OpenDecider-small-td | 0.792 | 0.715 | 0.703 | 0.107 | 38 ms (L40S) |
| OpenDecider-small | 0.672 | 0.735 | 0.702 | 0.087 | 38 ms (L40S) |
| Laya, typed-decisions checkpoint | 0.766 | 0.570 | 0.702 | 0.162 | 21 ms |
| TypeSafe Jev 1.13 | 0.754 | 0.730 | 0.774 | 0.164 | 404 ms (API) |
| CLM-8B | 0.358 | 0.400 | 0.431 | 0.106 | ~35 ms |
| Claude Fable 5.1 | – | 0.840 | – | 0.064 | 4.3 s (API) |
| OpenDecider-medium-td | 0.788 | 0.765 | 0.725 | 0.110 | 214 ms (4× L40S) |
| OpenDecider-large-td | 0.801 | 0.750 | 0.718 | 0.083 | 440 ms (4× L40S) |
Automating only the confident decisions. Most deployments let the model act on its confident cases and send the rest to a person. On typed-decisions, the most confident half of OpenDecider's answers is 94–95% accurate, against 88% for Jev. On the general decisions Jev ranks its confidence better (0.86 vs 0.83 for OpenDecider-small on the confident half). Laya's confidence barely separates right from wrong answers there: about 0.55 at every threshold.
Limitations
- Jev is the strongest model on Laya's own battery (0.774), especially phishing (0.90 vs 0.63 for OpenDecider).
- Laya wins the five battery tasks it was trained on (0.81 vs 0.66 for OpenDecider-small), and it has a multilingual checkpoint. OpenDecider is evaluated in English only.
- The general set is 200 items (about ±3 points), and the results are single training seeds.
- CLM-8B is a contrastive reranker rather than a classifier: near chance on typed-decisions and 0.0 on label-only BANKING77, and its model card's example (billing 0.939) did not reproduce through its official vLLM path (0.989).
Reproduce every number
git clone https://github.com/manjunathshiva/opendecider && cd opendecider
pip install -r benchmarks/requirements.txt
python benchmarks/report.py # every table, from the committed predictions
python benchmarks/run.py --model opendecider-nano # re-score a model yourself
The harness rebuilds all test items from their public sources (checked against SHA-256 manifests), commits only item ids and probabilities, and can score Laya, Jev (with your own key) or any OpenDecider folder.
What's next
opendecider-nano is coming to the browser: an ONNX build that runs on WebGPU or WebAssembly, so a web page decides without a server. After that: more than 26 options for the Qwen-based models, a rule-labelled evaluation, and a guide for fine-tuning nano on your own labels. Issues and benchmark requests are welcome on GitHub.
Issues and benchmark requests are welcome on GitHub.
Links: GitHub · Models · Live demo · Colab · Full comparison