Athekunal/ultrabert-jev-380m-sigmoid-norm

A decision model: it reads a state, a question and a list of options, and picks one option in a single forward pass, without generating text. It is UltraBERT-380M fully fine-tuned with one mask token in front of each option. Option k is scored from the masked-token logits at its mask.

Recipe: sigmoid of " yes" minus " no", divided by the sum over options. Trained for one epoch (batch 32, learning rate 1.5e-5) on the 13-dataset Reason+Long mixture of LLM-as-Jev, with Nimble added.

This checkpoint Accuracy (mean over 13 datasets)
Dev split 72.8%
Test split 71.9%

Usage

Load the weights in fp32. The masked-token logits are large (around 200), and bf16 weights changed accuracy by up to 8 points on long documents (MuSiQue) in our tests. All reported numbers use fp32 weights.

Use the prompt below exactly. The model was trained on this format with the tokens " yes" and " no"; plain fill-mask use will not give these results.

Tested with transformers 5.20.2 and torch 2.9, which include the UltraBERT model class. Do not pass trust_remote_code=True: the base repo's remote code names the weights differently and would leave them unloaded.

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

REPO = "Athekunal/ultrabert-jev-380m-sigmoid-norm"

tok = AutoTokenizer.from_pretrained(REPO)
# Load the weights in fp32. bf16 weights change the results (up to 8 points on MuSiQue).
model = AutoModelForMaskedLM.from_pretrained(REPO, dtype=torch.float32)
model = model.to("cuda").eval()

YES = tok(" yes", add_special_tokens=False)["input_ids"][0]
NO = tok(" no", add_special_tokens=False)["input_ids"][0]


def build_prompt(state: list[str], question: str, options: list[str]) -> str:
    """The exact prompt the model was trained on: one mask in front of each option."""
    text = "State: " + "".join(part + "\n\n" for part in state) + question + "\n"
    text += "Mark the correct option with yes and every other option with no.\nOptions:\n"
    text += "\n".join(f"[{k}] {tok.mask_token} {o}" for k, o in enumerate(options, 1))
    return text


@torch.no_grad()
def decide(state: list[str], question: str, options: list[str], score: str = "yes-no"):
    """(index of the chosen option, scores s_k). This checkpoint uses score="yes-no"."""
    enc = tok(build_prompt(state, question, options), return_tensors="pt").to(model.device)
    logits = model(**enc).logits[0]
    masks = (enc["input_ids"][0] == tok.mask_token_id).nonzero()[:, 0]
    s = logits[masks, YES] if score == "yes" else logits[masks, YES] - logits[masks, NO]
    return int(s.argmax()), s


choice, s = decide(
    state=["The parcel left the depot on Monday and was signed for on Wednesday."],
    question="Was the parcel delivered?",
    options=["Yes", "No"],
)
print(choice, s)

The decision is the option with the highest score s, for every recipe. state is a list of passages (for example a contract, or a context and a set of facts); each is followed by a blank line.

Related checkpoints

All four recipes share the data, the settings and the prompt; they differ in the score and the loss.

Repo Score Loss
ultrabert-jev-380m-softmax-yes " yes" logit softmax over the options
ultrabert-jev-380m-softmax-yesno " yes" βˆ’ " no" softmax over the options
ultrabert-jev-380m-sigmoid-norm " yes" βˆ’ " no" sigmoid of each score, divided by the sum
ultrabert-jev-380m-sqrt-softplus " yes" βˆ’ " no" square-root softplus of each score, divided by the sum

Evidence head

evidence_head/ holds a small head trained on this frozen encoder. It gives each token of the state an evidence score between 0 and 1 for the chosen option: how likely the token is to be part of the evidence an annotator marked. Positive tokens were up-weighted in training, so the score is not a calibrated probability. It was trained on human-marked evidence (ContractNLI, MuSiQue, Nimble, ProofWriter) and on generated labels selected with a reward adapted from SelfCite. It predicts annotated evidence; it is not an explanation of how the model reached its decision. evidence_head/threshold.json holds the score above which a word counts as marked, tuned on dev rows.

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from torch import nn


class EvidenceHead(nn.Module):
    """MLP over [h_t ; h_k ; h_t * h_k]: token t's hidden state and that of option k's mask."""

    def __init__(self, hidden: int = model.config.hidden_size, width: int = 512):
        super().__init__()
        self.mlp = nn.Sequential(nn.Linear(3 * hidden, width), nn.GELU(), nn.Dropout(0.1), nn.Linear(width, 1))

    def forward(self, tokens, answer):
        a = answer.expand_as(tokens)
        return self.mlp(torch.cat([tokens, a, tokens * a], -1))[..., 0]


head = EvidenceHead().to(model.device).eval()
head.load_state_dict(load_file(hf_hub_download(REPO, "evidence_head/model.safetensors")))


@torch.no_grad()
def evidence(state: list[str], question: str, options: list[str], answer: int | None = None):
    """[(part index, character start, character end, score)] for each token of the state."""
    text = build_prompt(state, question, options)
    enc = tok(text, return_tensors="pt", return_offsets_mapping=True)
    offsets = enc.pop("offset_mapping")[0].tolist()
    enc = enc.to(model.device)
    hidden = model.model(**enc).last_hidden_state[0]
    masks = (enc["input_ids"][0] == tok.mask_token_id).nonzero()[:, 0]
    if answer is None:
        answer, _ = decide(state, question, options)
    out, start = [], len("State: ")
    for i, part in enumerate(state):
        idx = [t for t, (a, b) in enumerate(offsets) if a < start + len(part) and b > start and b > a]
        p = torch.sigmoid(head(hidden[idx], hidden[masks[answer]])).tolist()
        out += [(i, max(offsets[t][0], start) - start, min(offsets[t][1], start + len(part)) - start, q)
                for t, q in zip(idx, p)]
        start += len(part) + 2
    return out

A sentence's score is the highest score among its tokens.

Limitations

  • Trained and evaluated in English, on one training mixture; accuracy outside it is lower (about 58–60% on six external benchmarks such as MMLU and BoolQ).
  • Inputs were at most 10,240 tokens in training.
  • The model gives no reasons for its choice.

training_config.yaml holds the full training configuration.

Downloads last month
77
Safetensors
Model size
0.4B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Athekunal/ultrabert-jev-380m-sigmoid-norm

Finetuned
(28)
this model

Papers for Athekunal/ultrabert-jev-380m-sigmoid-norm