OpenDecider: Open, Calibrated Decision Models That Beat Laya and Jev on Typed-Decisions

Community Article
Published September 27, 2026

OpenDecider vs TypeSafe Jev, Laya, CLM-8B and frontier LLMs, same questions and same scorer

Decision models ("System One" models) answer bounded questions about an input and return a probability for every allowed answer: which team should handle this ticket, how urgent is it, is this email phishing. TypeSafe's Jev made the interface popular as a closed API; Laya followed as an open model. This article introduces OpenDecider, a family of open decision models, and reports how it compares with Jev, Laya, Stanford and NVIDIA's CLM-8B and frontier LLMs when all of them answer the same questions and are scored by the same code.

The short version. OpenDecider-nano (~400M) scores 0.796 on typed-decisions, against 0.766 for Laya's fine-tuned checkpoint (paired bootstrap 95% CI for the gap: +0.014 to +0.044) and 0.754 for Jev measured through TypeSafe's own API. OpenDecider-small (4B) matches Jev on 200 general decisions none of these models trained on (0.735 vs 0.730) and is the best-calibrated self-hostable model I measured. It is not better everywhere: Jev leads Laya's own application battery, and phishing is OpenDecider's weakest task. Everything, including every logged answer from every system, is public.

Update, October 3, 2026 (0.6.0). Since this article was published:

  • Two larger models. opendecider-medium-td (Qwen3-30B-A3B) scores 0.765 on the 200 general decisions, ahead of Jev's 0.730, the best of any model you can run yourself. opendecider-large-td (Qwen3-Next-80B-A3B) has the best calibration (ECE 0.083) and is the closest to human judgement (ChaosNLI JSD 0.030). Both need NVIDIA GPUs.
  • Ollama and LM Studio: GGUF builds of small and small-td (Q8_0 and Q4_K_M).
  • A prompt guard against jailbreaks and prompt injection, in each agent framework's own hook (0.5.0).
  • TypeScript: npm i @opendecider/client, with tools for the Vercel AI SDK and Mastra (0.6.0). For Python apps without PyTorch: pip install opendecider-client.
  • Self-hosting: opendecider serve, a drop-in for Jev's /v1/systemone API, as a signed Docker image for x86 and ARM.

The original measurements below are unchanged; the new models are added to the tables.

Try it

pip install opendecider               # nano
pip install "opendecider[small]"      # the 4B models
pip install "opendecider[mlx]"        # Apple Silicon MLX builds
from opendecider import load

model = load("manjunathshiva/opendecider-nano")
result = model.system_one(
    "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan.",
    {
        "department": {"type": "choice", "instructions": "Which department should handle this?",
                       "criteria": {"billing": "invoices, payments, refunds",
                                    "technical": "bugs, outages, system errors",
                                    "other": "everything else"}},
        "urgency": {"type": "score", "instructions": "How urgent is this?",
                    "criteria": ["not urgent", "soon", "blocking"]},
        "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"},
    })
print(result["answers"]["department"]["choice"])   # billing        (0.927)
print(result["answers"]["urgency"]["score"])       # 2 = blocking   (0.604)
print(result["answers"]["churn_risk"]["noul"])     # 0.922

No install needed for a first look: the live demo runs nano on a CPU Space, and a Colab notebook runs both models on a free GPU (tested top to bottom on an NVIDIA GPU; small switches to fp16 on a T4).

The models

model size memory best for
opendecider-nano ~400M (Ettin encoder) 2.0 GiB speed: 16 ms per question on an L40S, ~4 ms batched; business workflow decisions
opendecider-small 4B (Qwen3-4B + LoRA) 8.9 GiB, tested on a 16 GB Mac mini decisions it has never seen: best accuracy and calibration
opendecider-small-td 4B 8.9 GiB business workflows like typed-decisions': 0.792
opendecider-small-mlx-8bit 4B, MLX 8-bit 4.5 GB Macs: same top answer as full precision on 1,955 of 2,000 typed-decisions questions, 66 ms
opendecider-small-mlx-4bit 4B, MLX 4-bit 2.6 GB Macs with little memory; about 2 points lower on typed-decisions
opendecider-medium-td 30B (3B active), Qwen3-30B-A3B + LoRA 61 GB bf16, NVIDIA (tested on 4× L40S) general decisions: 0.765, the best self-hostable model; closest to human judgement
opendecider-large-td 80B (3B active), Qwen3-Next-80B-A3B + LoRA 160 GB bf16, NVIDIA (tested on 4× L40S) thresholding on probabilities: best calibration
opendecider-small-GGUF 4B, GGUF Q8_0 / Q4_K_M 4.3 / 2.5 GB LM Studio and Ollama; typed-decisions 0.669 at Q8_0 (full precision 0.671)
opendecider-small-td-GGUF 4B, GGUF Q8_0 / Q4_K_M 4.3 / 2.5 GB LM Studio and Ollama, business workflows: 0.794 at Q8_0 (full precision 0.792)

All are Apache-2.0 and give identical answers on Apple Silicon (MPS), NVIDIA (CUDA) and CPU.

How they work

OpenDecider-nano reads question: …, [MASK] option 1, [MASK] option 2, …, input: <state> in one encoder pass. The hidden state at each [MASK] goes through a small MLP to one logit, then a softmax across the question's options. This option-marker idea is the same one Laya uses; the difference is that there is no fixed token budget shared by the options, so a 78-option question still gets every option in full. That is why nano scores 0.645 on 77-label BANKING77 where Laya scores 0.425.

OpenDecider-small is Qwen3-4B-Instruct-2507 with a LoRA adapter. The options are lettered, and one forward pass reads the probability of each letter as the next token; above 26 options it scores each option name's log-likelihood.

Both are trained by distillation from calibrated teachers. Two openly licensed teachers, Qwen3-235B-A22B-Instruct-2507 (Apache-2.0) and DeepSeek V4.1 Flash (MIT), answered every training question with lettered options, and their option distributions were read from next-token log-probabilities. Each teacher was temperature-scaled on held-out gold labels before the two were averaged, so the students learn calibrated distributions rather than hard labels. The effect shows directly: untrained Qwen3-4B has an expected calibration error of 0.289; OpenDecider-small has 0.087.

The training data (~190K questions) combines public classification, NLI, reading-comprehension, topic, toxicity, spam, relevance and paraphrase datasets with synthetic business cases, emails and product reviews. Every dataset used for evaluation, and its family, is excluded, and every training pool was checked for text overlap with all test sets: 0 overlaps. nano and small-td then had a short fine-tune on the typed-decisions train split, like Laya's typed-decisions checkpoint; the test split was never used for training or model selection. Training ran on AWS SageMaker (NVIDIA L40S) for under $30 of GPU time.

How they compare

Every system answered the same questions and was scored by the same code:

  • Jev through TypeSafe's own API with native typed questions, not published figures.
  • typed-decisions scored with the Jev-vs-Laya harness published by Kameshwara Pavan kumar Mantha and the Antz AI team, joined question by question with their results (all 2,000 gold labels matched).
  • Laya's application battery run through Laya's own bench_apps.py, unchanged, at a pinned commit.
  • CLM-8B run with its own engine through the official vLLM pooling server.
typed-decisions (2,000) 200 general decisions Laya battery (10 × 400) calibration error ↓ latency
OpenDecider-nano 0.796 0.680 0.656 0.092 16 ms (L40S)
OpenDecider-small-td 0.792 0.715 0.703 0.107 38 ms (L40S)
OpenDecider-small 0.672 0.735 0.702 0.087 38 ms (L40S)
Laya, typed-decisions checkpoint 0.766 0.570 0.702 0.162 21 ms
TypeSafe Jev 1.13 0.754 0.730 0.774 0.164 404 ms (API)
CLM-8B 0.358 0.400 0.431 0.106 ~35 ms
Claude Fable 5.1 – 0.840 – 0.064 4.3 s (API)
OpenDecider-medium-td 0.788 0.765 0.725 0.110 214 ms (4× L40S)
OpenDecider-large-td 0.801 0.750 0.718 0.083 440 ms (4× L40S)

Automating only the confident decisions. Most deployments let the model act on its confident cases and send the rest to a person. On typed-decisions, the most confident half of OpenDecider's answers is 94–95% accurate, against 88% for Jev. On the general decisions Jev ranks its confidence better (0.86 vs 0.83 for OpenDecider-small on the confident half). Laya's confidence barely separates right from wrong answers there: about 0.55 at every threshold.

Limitations

  • Jev is the strongest model on Laya's own battery (0.774), especially phishing (0.90 vs 0.63 for OpenDecider).
  • Laya wins the five battery tasks it was trained on (0.81 vs 0.66 for OpenDecider-small), and it has a multilingual checkpoint. OpenDecider is evaluated in English only.
  • The general set is 200 items (about ±3 points), and the results are single training seeds.
  • CLM-8B is a contrastive reranker rather than a classifier: near chance on typed-decisions and 0.0 on label-only BANKING77, and its model card's example (billing 0.939) did not reproduce through its official vLLM path (0.989).

Reproduce every number

git clone https://github.com/manjunathshiva/opendecider && cd opendecider
pip install -r benchmarks/requirements.txt
python benchmarks/report.py                               # every table, from the committed predictions
python benchmarks/run.py --model opendecider-nano         # re-score a model yourself

The harness rebuilds all test items from their public sources (checked against SHA-256 manifests), commits only item ids and probabilities, and can score Laya, Jev (with your own key) or any OpenDecider folder.

What's next

opendecider-nano is coming to the browser: an ONNX build that runs on WebGPU or WebAssembly, so a web page decides without a server. After that: more than 26 options for the Qwen-based models, a rule-labelled evaluation, and a guide for fine-tuning nano on your own labels. Issues and benchmark requests are welcome on GitHub.

Issues and benchmark requests are welcome on GitHub.

Links: GitHub · Models · Live demo · Colab · Full comparison

Community

Sign up or log in to comment