Finding diseases in Spanish clinical text: two open models, one sealed test

Antecedentes: HTA, DM2, EPOC e IC. Ingresa por una reagudización de EPOC.

That is one line of a Spanish patient history, written for our model cards. It names four diseases and spells out none of them. We gave it to our two new open models for Spanish clinical text, and both returned all five mentions (HTA, DM2, EPOC, IC and the second EPOC) with exact character offsets. Both scored IC, the shortest, below HTA, DM2 and EPOC: 0.58 from one model, 0.62 from the other.

On another line, they disagree. On "El cribado neonatal detectó fibrosis quística, hipotiroidismo congénito y fenilcetonuria", the GLiNER2.5 model returns two diseases and the mmBERT model returns all three.

This post is about those two models: OpenMed GLiNER2.5 ClinicalNER Disease Spanish and OpenMed mmBERT ClinicalNER Disease Spanish. How we built them, how we tested them on case reports they never saw, where each one goes wrong, and which one to pick. Both are open weights under Apache-2.0 and run on a CPU.

Why Spanish clinical text needs its own model

Spanish clinical writing is dense. Histories lean on abbreviations (HTA, DM2, EPOC, IC). Disease names are long noun phrases where the boundary is the whole point: poliquistosis renal autosómica dominante, linfoma difuso de células B grandes. And diseases show up inside negations and family history as often as in the diagnosis line.

A strong multilingual extractor can already be pointed at Spanish with a prompt. We started from Fastino's GLiNER2.5 multi with the Spanish label enfermedad o trastorno. On our 150 held-out case reports, at its best threshold, it found 690 of 1,723 annotated disease mentions and returned 1,119 spans that matched none: F1 39.1. That is a fair start for a model that never saw this annotation style, and it is the reference we had to beat.

The data: 750 case reports, split by publication before training

We used DisTEMIST, a corpus of Spanish clinical case reports from the medical literature with human-annotated disease mentions, released under CC BY 4.0. We downloaded only its original training shard: 750 case reports with 8,065 disease spans. We never opened the official DisTEMIST test set, so our numbers are not comparable to the shared-task leaderboard.

Several case reports can come from the same publication, and near-identical text can leak between splits. So we grouped reports by publication (709 groups), checked for exact and near-duplicate text, and fixed the split by group before any training:

Nested and crossing mentions stay in the gold set and count in every score. The original text is never normalized, so offsets point into the real string.

How we built them

GLiNER2.5 Spanish starts from fastino/gliner2.5-multi-v1 (287M parameters). We kept its native span architecture and fine-tuned it with the GLiNER2 trainer on one label prompt, enfermedad o trastorno. It returns spans with a confidence score, so it has a threshold.

mmBERT Spanish starts from jhu-clsp/mmBERT-base (308M), a multilingual encoder from Johns Hopkins CLSP. We added a new token-classification head with BIO tags. Flat tags cannot hold a mention nested inside another, so 123 nested or crossing spans were dropped from its training windows. Evaluation still counts every one.

Both used the same recipe:

• 1,000-character windows with 300 characters of overlap, at most 512 tokens
• 8 epochs, batch size 4, encoder learning rate 2e-5, task or head learning rate 1e-4, seed 20261030
• a tuning-set check after every second epoch
• 21 of 1,539 training windows skipped for GLiNER2.5 and 17 for mmBERT, where a gold span did not line up with token boundaries

How we tested: tune, lock, then open the held-out set once

Every choice was made on the 148 tuning documents. GLiNER2.5 peaked at epoch 4 with threshold 0.4 (73.4 tuning F1). mmBERT kept improving and was picked at epoch 8 (67.8).

mmBERT's new head starts out random (0.5 F1 on the tuning set), so it is not a fair baseline. Both models are compared to the untuned GLiNER2.5 multi, which can already extract diseases from a prompt.

We also made that reference stronger before the test. Its best tuning threshold sat at the edge of our grid (0.7), so we widened its grid to 0.95 and tried the English label disease as well. The Spanish prompt at 0.7 stayed best (36.9 tuning F1, against 36.3 for the English label).

That chart holds a useful surprise: after fine-tuning, the threshold barely moves F1. It becomes a clean precision and recall dial instead. On the tuning set, 0.7 gives 77.3 precision and 69.1 recall; 0.1 gives 71.4 precision and 74.4 recall.

Then we hashed the selection, wrote down the pass rules and opened the 150 held-out case reports once. A model passed only if:

1. the 95% interval of its F1 gain over the untuned model, from a paired bootstrap over publication groups, sat above zero;
2. its recall did not drop by more than half a point;
3. the held-out set covered at least 100 publication groups.

Results on 150 held-out case reports

Exact-span scores: a prediction counts only if its start and end match the annotation to the character.

Both models passed all three rules. The intervals come from 2,000 resamples of the 142 publication groups, so documents from the same paper move together.

Each model runs at its own threshold, chosen before the test. At one shared threshold, 0.5, GLiNER2.5 Spanish scores 73.08 F1 and the untuned model 36.01.

Where the errors are

F1 hides the shape of the mistakes, so here are the counts behind it.

• mmBERT finds 18 more mentions (1,259 against 1,241) and misses 18 fewer.
• GLiNER2.5 raises 193 fewer false alarms (424 against 617).
• Most of the F1 gap between them is false alarms, not misses.

Under exact matching, a boundary slip costs twice. If a model returns insuficiencia renal where the annotators marked insuficiencia renal aguda, that is one miss and one false alarm. Our reports count errors; they do not yet split boundary slips from spans that are wrong outright, so we cannot tell you which kind dominates.

The newborn-screening line at the top is the same trade-off in one sentence. But one sentence is not a benchmark: across 150 documents, the recall gap is 18 mentions.

Which one to pick

Pick GLiNER2.5 Spanish when every span you return costs someone's review time: problem lists, cohort screening with a human check, coding support. It is the more precise of the two, and the threshold gives you a dial.

Pick mmBERT Spanish when a missed mention costs more than an extra one to review, or when you already run Transformers token classifiers. Treat its scores as a ranking, not a probability: 30 of 32 spans in its card examples round to 1.00.

What it is not

• Not tested on hospital notes. Training and testing used Spanish case reports published in the medical literature. The notes in this post are synthetic.
• Mentions, not diagnoses. Sin antecedentes de asma ni de epilepsia still returns asma and epilepsia. A ruled-out síndrome coronario agudo is still tagged, and so is the mother's enfermedad de Huntington. OpenMed's assert_context rules are written for English and have not been validated on Spanish, so check negation and family history yourself.
• No clean-document test. Every source document mentions a disease, so we have not measured how often the models fire on a note that has none.
• mmBERT returns one flat sequence. Where the annotators marked a mention inside another, it returns one of them.
• No codes. Spans are not mapped to SNOMED CT or ICD-10.
• Not a leaderboard claim. We did not use the official DisTEMIST test set and make no state-of-the-art claim. Existing DisTEMIST-trained models were trained on this same original training partition, so their scores on our held-out set would not be a fair comparison.
• "Unseen" means unseen in our program. Whether the base models' pretraining saw these published case reports is unknown.
• Not a medical device. The models find text spans. They do not diagnose anything, and they are not de-identification tools.

By design, symptoms are not diseases (disnea, ortopnea and edemas are not returned, following the DisTEMIST guidelines), and drug names return nothing.

Try it

Both models run through analyze_text in the OpenMed Python SDK (Apache-2.0). Version 3.2.0 is on PyPI.

pip install -U "openmed[gliner2]"  # OpenMed 3.2 or later

from openmed import analyze_text

text = (
    "Varón de 67 años con diabetes mellitus tipo 2, "
    "enfermedad renal crónica estadio 3 y fibrilación "
    "auricular, que ingresa por una neumonía adquirida en "
    "la comunidad."
)
result = analyze_text(
    text,
    model_id="OpenMed/GLiNER2.5-ClinicalNER-Disease-Spanish-v1",
    labels=["enfermedad o trastorno"],
    threshold=0.4,
    sentence_language="es",
)
for e in result.entities:
    print(e.start, e.end, e.text, f"{e.confidence:.2f}")

21 45 diabetes mellitus tipo 2 0.99
47 71 enfermedad renal crónica 0.96
84 105 fibrilación auricular 0.98
127 135 neumonía 0.94

For mmBERT, install openmed[hf] and drop the label and threshold:

result = analyze_text(
    text,
    model_id="OpenMed/mmBERT-ClinicalNER-Disease-Spanish-Base-v1",
    sentence_language="es",
)

Pass a whole note: analyze_text splits it into sentences and returns offsets into the original text. The model loads once and stays cached, so loop over many notes. On a GPU, pass config=OpenMedConfig(device="cuda").

Thank you

These models stand on open work. Thank you to Fastino for GLiNER2.5 multi and the open-source GLiNER2 library, to Johns Hopkins University CLSP for mmBERT, and to Antonio Miranda-Escalada and the DisTEMIST authors for a carefully annotated Spanish corpus released under CC BY 4.0. Spanish clinical NLP is in good shape because of people who share like this.

• OpenMed GLiNER2.5 ClinicalNER Disease Spanish v1
• OpenMed mmBERT ClinicalNER Disease Spanish Base v1
• DisTEMIST corpus (Zenodo)
• All OpenMed models