Instructions to use Athekunal/ultrabert-jev-380m-sigmoid-norm with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Athekunal/ultrabert-jev-380m-sigmoid-norm with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="Athekunal/ultrabert-jev-380m-sigmoid-norm")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("Athekunal/ultrabert-jev-380m-sigmoid-norm") model = AutoModelForMaskedLM.from_pretrained("Athekunal/ultrabert-jev-380m-sigmoid-norm", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Athekunal/ultrabert-jev-380m-sigmoid-norm
A decision model: it reads a state, a question and a list of options, and picks one option in a single forward pass, without generating text. It is UltraBERT-380M fully fine-tuned with one mask token in front of each option. Option k is scored from the masked-token logits at its mask.
Recipe: sigmoid of " yes" minus " no", divided by the sum over options. Trained for one epoch (batch 32, learning rate 1.5e-5) on the 13-dataset Reason+Long mixture of LLM-as-Jev, with Nimble added.
| This checkpoint | Accuracy (mean over 13 datasets) |
|---|---|
| Dev split | 72.8% |
| Test split | 71.9% |
Usage
Load the weights in fp32. The masked-token logits are large (around 200), and bf16 weights changed accuracy by up to 8 points on long documents (MuSiQue) in our tests. All reported numbers use fp32 weights.
Use the prompt below exactly. The model was trained on this format with the tokens
" yes"and" no"; plain fill-mask use will not give these results.
Tested with transformers 5.20.2 and torch 2.9, which include the UltraBERT model class. Do not pass trust_remote_code=True: the base repo's remote code names the weights differently and would leave them unloaded.
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer
REPO = "Athekunal/ultrabert-jev-380m-sigmoid-norm"
tok = AutoTokenizer.from_pretrained(REPO)
# Load the weights in fp32. bf16 weights change the results (up to 8 points on MuSiQue).
model = AutoModelForMaskedLM.from_pretrained(REPO, dtype=torch.float32)
model = model.to("cuda").eval()
YES = tok(" yes", add_special_tokens=False)["input_ids"][0]
NO = tok(" no", add_special_tokens=False)["input_ids"][0]
def build_prompt(state: list[str], question: str, options: list[str]) -> str:
"""The exact prompt the model was trained on: one mask in front of each option."""
text = "State: " + "".join(part + "\n\n" for part in state) + question + "\n"
text += "Mark the correct option with yes and every other option with no.\nOptions:\n"
text += "\n".join(f"[{k}] {tok.mask_token} {o}" for k, o in enumerate(options, 1))
return text
@torch.no_grad()
def decide(state: list[str], question: str, options: list[str], score: str = "yes-no"):
"""(index of the chosen option, scores s_k). This checkpoint uses score="yes-no"."""
enc = tok(build_prompt(state, question, options), return_tensors="pt").to(model.device)
logits = model(**enc).logits[0]
masks = (enc["input_ids"][0] == tok.mask_token_id).nonzero()[:, 0]
s = logits[masks, YES] if score == "yes" else logits[masks, YES] - logits[masks, NO]
return int(s.argmax()), s
choice, s = decide(
state=["The parcel left the depot on Monday and was signed for on Wednesday."],
question="Was the parcel delivered?",
options=["Yes", "No"],
)
print(choice, s)
The decision is the option with the highest score s, for every recipe. state is a list of passages (for example a contract, or a context and a set of facts); each is followed by a blank line.
Related checkpoints
All four recipes share the data, the settings and the prompt; they differ in the score and the loss.
| Repo | Score | Loss |
|---|---|---|
| ultrabert-jev-380m-softmax-yes | " yes" logit |
softmax over the options |
| ultrabert-jev-380m-softmax-yesno | " yes" β " no" |
softmax over the options |
| ultrabert-jev-380m-sigmoid-norm | " yes" β " no" |
sigmoid of each score, divided by the sum |
| ultrabert-jev-380m-sqrt-softplus | " yes" β " no" |
square-root softplus of each score, divided by the sum |
Evidence head
evidence_head/ holds a small head trained on this frozen encoder. It gives each token of the state an evidence score between 0 and 1 for the chosen option: how likely the token is to be part of the evidence an annotator marked. Positive tokens were up-weighted in training, so the score is not a calibrated probability.
It was trained on human-marked evidence (ContractNLI, MuSiQue, Nimble, ProofWriter) and on generated labels selected with a reward adapted from SelfCite.
It predicts annotated evidence; it is not an explanation of how the model reached its decision.
evidence_head/threshold.json holds the score above which a word counts as marked, tuned on dev rows.
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from torch import nn
class EvidenceHead(nn.Module):
"""MLP over [h_t ; h_k ; h_t * h_k]: token t's hidden state and that of option k's mask."""
def __init__(self, hidden: int = model.config.hidden_size, width: int = 512):
super().__init__()
self.mlp = nn.Sequential(nn.Linear(3 * hidden, width), nn.GELU(), nn.Dropout(0.1), nn.Linear(width, 1))
def forward(self, tokens, answer):
a = answer.expand_as(tokens)
return self.mlp(torch.cat([tokens, a, tokens * a], -1))[..., 0]
head = EvidenceHead().to(model.device).eval()
head.load_state_dict(load_file(hf_hub_download(REPO, "evidence_head/model.safetensors")))
@torch.no_grad()
def evidence(state: list[str], question: str, options: list[str], answer: int | None = None):
"""[(part index, character start, character end, score)] for each token of the state."""
text = build_prompt(state, question, options)
enc = tok(text, return_tensors="pt", return_offsets_mapping=True)
offsets = enc.pop("offset_mapping")[0].tolist()
enc = enc.to(model.device)
hidden = model.model(**enc).last_hidden_state[0]
masks = (enc["input_ids"][0] == tok.mask_token_id).nonzero()[:, 0]
if answer is None:
answer, _ = decide(state, question, options)
out, start = [], len("State: ")
for i, part in enumerate(state):
idx = [t for t, (a, b) in enumerate(offsets) if a < start + len(part) and b > start and b > a]
p = torch.sigmoid(head(hidden[idx], hidden[masks[answer]])).tolist()
out += [(i, max(offsets[t][0], start) - start, min(offsets[t][1], start + len(part)) - start, q)
for t, q in zip(idx, p)]
start += len(part) + 2
return out
A sentence's score is the highest score among its tokens.
Limitations
- Trained and evaluated in English, on one training mixture; accuracy outside it is lower (about 58β60% on six external benchmarks such as MMLU and BoolQ).
- Inputs were at most 10,240 tokens in training.
- The model gives no reasons for its choice.
training_config.yaml holds the full training configuration.
- Downloads last month
- 77
Model tree for Athekunal/ultrabert-jev-380m-sigmoid-norm
Base model
dmis-lab/UltraBERT-380M