MattStammers/BioClinicalBERT_IBD

Model Card for BioClinicalBERT IBD

The model classifies documents as either IBD or Not IBD

Model Details

Model Description

As above. This is a model trained to detect IBD patients from clinical text

• Developed by: Matt Stammers
• Funded by: University Hospital Foundation NHS Trust
• Shared by: Matt Stammers - SETT Data and AI Clinical Lead
• Model type: BERT Transformer
• Language(s) (NLP): English
• License: cc-by-nc-4.0
• Finetuned from model: emilyalsentzer/Bio_ClinicalBERT

Model Sources

• Repository: https://huggingface.co/MattStammers/BioClinicalBERT_IBD
• Paper: Arxiv (Pending)
• Demo: https://huggingface.co/spaces/MattStammers/IBD_Cohort_Identification

Uses

For document classification tasks to differentiate between documents likely to be diagnostic of IBD and those unlikely to be diagnostic of IBD.

Direct Use

This model can be used directly at Cohort Identification Demo

Downstream Use

Others can build on this model and improve it but only for non-commercial purposes.

Out-of-Scope Use

This model is less powerful (in terms of F1 Score) when making predictions at the patient level by 1-2%. It can be used for this purpose but with care.

Bias, Risks, and Limitations

This model contains substantial biases and is known to be biased against older patients and non-white patients so use with care.

Recommendations

It will work best in a predominantly younger caucasian population.

How to Get Started with the Model

Use the code below to get started with the model.

The model is best used with the transformers library.

Training Details

Training Data

The model was trained on fully pseudonymised clinical information at UHSFT which was carefully labelled by a consultant (attending) physician and evaluated against a randomly selected internal holdout set.

Training Procedure

See the paper for more information on the training procedure

Training Hyperparameters

• Training regime: fp32

Speeds, Sizes, Times

This model (part of a set of models) took 213.55 minutes to train

Evaluation

The model was internally validated against a holdout set

Testing Data, Factors & Metrics

Testing Data

The testing data cannot be revealed due to IG regulations and to remain compliant with GDPR, only the resulting model can be

Factors

IBD vs Not-IBD

Metrics

Full evaluation metrics are available in the paper with a summary below

Results

Model	Doc Coverage	Accuracy	Precision	Recall	Specificity	NPV	F1 Score	MCC
BioclinicalBERT	768 (100.00%)	90.29% (CI: 87.33% - 92.62%)	91.48% (CI: 88.39% - 93.81%)	96.91% (CI: 94.67% - 98.22%)	63.54% (CI: 53.57% - 72.48%)	83.56% (CI: 73.43% - 90.34%)	94.12% (CI: 92.79% - 95.48%)	0.6735 (CI: 0.5892 - 0.7538)

Summary

Overall performance of the model is high with an F1 Score of >94% on our internal holdout set.

Environmental Impact

Training the model used 2.01kWh of energy emmitting 416.73 grams of CO2

• Hardware Type: L40S
• Hours used: 3.55
• Carbon Emitted: 0.417 Kg CO2

Citation

Arxiv (Pending)

Glossary

Term	Description
Accuracy	The percentage of results that were correct among all results from the system. Calc: (TP + TN) / (TP + FP + TN + FN).
Precision (PPV)	Also called positive predictive value (PPV), it is the percentage of true positive results among all results that the system flagged as positive. Calc: TP / (TP + FP).
Negative Predictive Value (NPV)	The percentage of results that were true negative (TN) among all results that the system flagged as negative. Calc: TN / (TN + FN).
Recall	Also called sensitivity. The percentage of results flagged positive among all results that should have been obtained. Calc: TP / (TP + FN).
Specificity	The percentage of results that were flagged negative among all negative results. Calc: TN / (TN + FP).
F1-Score	The harmonic mean of PPV/precision and sensitivity/recall. Calc: 2 × (Precision × Recall) / (Precision + Recall). Moderately useful in the context of class imbalance.
Matthews’ Correlation Coefficient (MCC)	A statistical measure used to evaluate the quality of binary classifications. Unlike other metrics, MCC considers all four categories of a confusion matrix. Calc: (TP × TN − FP × FN) / √((TP + FP)(TP + FN)(TN + FP)(TN + FN)).
Precision / Recall AUC	Represents the area under the Precision-Recall curve, which plots Precision against Recall at various threshold settings. It is more resistant to class imbalance than alternatives like AUROC.
Demographic Parity (DP)	Demographic Parity, also known as Statistical Parity, requires that the probability of a positive prediction is the same across different demographic groups. Calc: DP = P(Ŷ=1∣A=a) = P(Ŷ=1∣A=b). This figure is given as an absolute difference where positive values suggest the more privileged group gains and negative values the reverse.
Equal Opportunity (EO)	Equal Opportunity focuses on equalising the true positive rates across groups. Among those who truly belong to the positive class, the model should predict positive outcomes at equal rates across groups. Calc: EO = P(Ŷ=1∣Y=1, A=a) = P(Ŷ=1∣Y=1, A=b). A higher value indicates a bias against the more vulnerable group.
Disparate Impact (DI)	Divides the protected group’s positive prediction rate by that of the most-favoured group. If the ratio is below 0.8 or above 1.25, disparate impact is considered present. Calc: DI = P(Ŷ=1∣A=unfavoured) / P(Ŷ=1∣A=favoured). Values outside 0.8–1.25 range suggest bias.
Execution Time / Energy / CO₂ Emissions	Measured in minutes and total energy consumption in kilowatt-hours (kWh), which is then converted to CO₂ emissions using a factor of 0.20705 Kg CO₂e per kWh.

Model Card Authors

Matt Stammers - Computational Gastroenterologist

Model Card Contact

[email protected]