Token Classification
Transformers
Safetensors
PyTorch
Japanese
qwen3
ner
pii
pii-detection
de-identification
privacy
healthcare
medical
clinical
phi
openmed
japanese
text-generation-inference
Instructions to use OpenMed/OpenMed-PII-Japanese-QwenMed-XLarge-600M-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use OpenMed/OpenMed-PII-Japanese-QwenMed-XLarge-600M-v1 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="OpenMed/OpenMed-PII-Japanese-QwenMed-XLarge-600M-v1")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForTokenClassification tokenizer = AutoTokenizer.from_pretrained("OpenMed/OpenMed-PII-Japanese-QwenMed-XLarge-600M-v1") model = AutoModelForTokenClassification.from_pretrained("OpenMed/OpenMed-PII-Japanese-QwenMed-XLarge-600M-v1", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Fix model card language metadata
Browse filesREADME-only language metadata correction. Repository settings are unchanged.
README.md
CHANGED
|
@@ -1,302 +1,74 @@
|
|
| 1 |
---
|
| 2 |
-
language:
|
| 3 |
-
- ar
|
| 4 |
license: apache-2.0
|
|
|
|
|
|
|
| 5 |
base_model: Qwen/Qwen3-Embedding-0.6B
|
| 6 |
-
tags:
|
| 7 |
-
- token-classification
|
| 8 |
-
- ner
|
| 9 |
-
- pii
|
| 10 |
-
- pii-detection
|
| 11 |
-
- de-identification
|
| 12 |
-
- privacy
|
| 13 |
-
- healthcare
|
| 14 |
-
- medical
|
| 15 |
-
- clinical
|
| 16 |
-
- phi
|
| 17 |
-
- arabic
|
| 18 |
-
- pytorch
|
| 19 |
-
- transformers
|
| 20 |
-
- openmed
|
| 21 |
-
pipeline_tag: token-classification
|
| 22 |
library_name: transformers
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
value: 0.7929
|
| 40 |
-
name: F1 (micro)
|
| 41 |
-
- type: precision
|
| 42 |
-
value: 0.7832
|
| 43 |
-
name: Precision
|
| 44 |
-
- type: recall
|
| 45 |
-
value: 0.8028
|
| 46 |
-
name: Recall
|
| 47 |
widget:
|
| 48 |
-
|
| 49 |
-
|
| 50 |
---
|
| 51 |
|
| 52 |
-
# OpenMed-PII-
|
| 53 |
-
|
| 54 |
-
**Arabic PII Detection Model** | 600M Parameters | Open Source
|
| 55 |
-
|
| 56 |
-
[]() []() []()
|
| 57 |
-
|
| 58 |
-
## Model Description
|
| 59 |
-
|
| 60 |
-
**OpenMed-PII-Arabic-QwenMed-XLarge-600M-v1** is a transformer-based token classification model fine-tuned for **Personally Identifiable Information (PII) detection in Arabic text**. This model identifies and classifies **54 types of sensitive information** including names, addresses, social security numbers, medical record numbers, and more.
|
| 61 |
-
|
| 62 |
-
### Key Features
|
| 63 |
-
|
| 64 |
-
- **Arabic-Optimized**: Specifically trained on Arabic text for optimal performance
|
| 65 |
-
- **High Accuracy**: Achieves strong F1 scores across diverse PII categories
|
| 66 |
-
- **Comprehensive Coverage**: Detects 55+ entity types spanning personal, financial, medical, and contact information
|
| 67 |
-
- **Privacy-Focused**: Designed for de-identification and compliance with GDPR and other privacy regulations
|
| 68 |
-
- **Production-Ready**: Optimized for real-world text processing pipelines
|
| 69 |
-
|
| 70 |
-
## Performance
|
| 71 |
-
|
| 72 |
-
Evaluated on the Arabic test split (AI4Privacy + synthetic data):
|
| 73 |
-
|
| 74 |
-
| Metric | Score |
|
| 75 |
-
|:---|:---:|
|
| 76 |
-
| **Micro F1** | **0.7929** |
|
| 77 |
-
| Precision | 0.7832 |
|
| 78 |
-
| Recall | 0.8028 |
|
| 79 |
-
| Macro F1 | 0.4080 |
|
| 80 |
-
| Weighted F1 | 0.8030 |
|
| 81 |
-
| Accuracy | 0.8920 |
|
| 82 |
-
|
| 83 |
-
### Top 3 Arabic PII Models
|
| 84 |
-
|
| 85 |
-
| Rank | Model | F1 | Precision | Recall |
|
| 86 |
-
|:---:|:---|:---:|:---:|:---:|
|
| 87 |
-
| 1 | [OpenMed-PII-Arabic-BigMed-Large-560M-v1](https://huggingface.co/OpenMed/OpenMed-PII-Arabic-BigMed-Large-560M-v1) | 0.8410 | 0.8531 | 0.8292 |
|
| 88 |
-
| 2 | [OpenMed-PII-Arabic-NomicMed-Large-395M-v1](https://huggingface.co/OpenMed/OpenMed-PII-Arabic-NomicMed-Large-395M-v1) | 0.8281 | 0.8126 | 0.8441 |
|
| 89 |
-
| **3** | **[OpenMed-PII-Arabic-QwenMed-XLarge-600M-v1](https://huggingface.co/OpenMed/OpenMed-PII-Arabic-QwenMed-XLarge-600M-v1)** | **0.7929** | **0.7832** | **0.8028** |
|
| 90 |
-
|
| 91 |
-
## Supported Entity Types
|
| 92 |
-
|
| 93 |
-
This model detects **54 PII entity types** organized into categories:
|
| 94 |
-
|
| 95 |
-
<details>
|
| 96 |
-
<summary><strong>Identifiers</strong> (22 types)</summary>
|
| 97 |
-
|
| 98 |
-
| Entity | Description |
|
| 99 |
-
|:---|:---|
|
| 100 |
-
| `ACCOUNTNAME` | Accountname |
|
| 101 |
-
| `BANKACCOUNT` | Bankaccount |
|
| 102 |
-
| `BIC` | Bic |
|
| 103 |
-
| `BITCOINADDRESS` | Bitcoinaddress |
|
| 104 |
-
| `CREDITCARD` | Creditcard |
|
| 105 |
-
| `CREDITCARDISSUER` | Creditcardissuer |
|
| 106 |
-
| `CVV` | Cvv |
|
| 107 |
-
| `ETHEREUMADDRESS` | Ethereumaddress |
|
| 108 |
-
| `IBAN` | Iban |
|
| 109 |
-
| `IMEI` | Imei |
|
| 110 |
-
| ... | *and 12 more* |
|
| 111 |
-
|
| 112 |
-
</details>
|
| 113 |
-
|
| 114 |
-
<details>
|
| 115 |
-
<summary><strong>Personal Info</strong> (11 types)</summary>
|
| 116 |
-
|
| 117 |
-
| Entity | Description |
|
| 118 |
-
|:---|:---|
|
| 119 |
-
| `AGE` | Age |
|
| 120 |
-
| `DATEOFBIRTH` | Dateofbirth |
|
| 121 |
-
| `EYECOLOR` | Eyecolor |
|
| 122 |
-
| `FIRSTNAME` | Firstname |
|
| 123 |
-
| `GENDER` | Gender |
|
| 124 |
-
| `HEIGHT` | Height |
|
| 125 |
-
| `LASTNAME` | Lastname |
|
| 126 |
-
| `MIDDLENAME` | Middlename |
|
| 127 |
-
| `OCCUPATION` | Occupation |
|
| 128 |
-
| `PREFIX` | Prefix |
|
| 129 |
-
| ... | *and 1 more* |
|
| 130 |
-
|
| 131 |
-
</details>
|
| 132 |
-
|
| 133 |
-
<details>
|
| 134 |
-
<summary><strong>Contact Info</strong> (2 types)</summary>
|
| 135 |
-
|
| 136 |
-
| Entity | Description |
|
| 137 |
-
|:---|:---|
|
| 138 |
-
| `EMAIL` | Email |
|
| 139 |
-
| `PHONE` | Phone |
|
| 140 |
-
|
| 141 |
-
</details>
|
| 142 |
-
|
| 143 |
-
<details>
|
| 144 |
-
<summary><strong>Location</strong> (9 types)</summary>
|
| 145 |
-
|
| 146 |
-
| Entity | Description |
|
| 147 |
-
|:---|:---|
|
| 148 |
-
| `BUILDINGNUMBER` | Buildingnumber |
|
| 149 |
-
| `CITY` | City |
|
| 150 |
-
| `COUNTY` | County |
|
| 151 |
-
| `GPSCOORDINATES` | Gpscoordinates |
|
| 152 |
-
| `ORDINALDIRECTION` | Ordinaldirection |
|
| 153 |
-
| `SECONDARYADDRESS` | Secondaryaddress |
|
| 154 |
-
| `STATE` | State |
|
| 155 |
-
| `STREET` | Street |
|
| 156 |
-
| `ZIPCODE` | Zipcode |
|
| 157 |
-
|
| 158 |
-
</details>
|
| 159 |
-
|
| 160 |
-
<details>
|
| 161 |
-
<summary><strong>Organization</strong> (3 types)</summary>
|
| 162 |
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
|
| 166 |
-
| `JOBTITLE` | Jobtitle |
|
| 167 |
-
| `ORGANIZATION` | Organization |
|
| 168 |
|
| 169 |
-
|
| 170 |
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
|
| 174 |
-
|
| 175 |
-
|:---|:---|
|
| 176 |
-
| `AMOUNT` | Amount |
|
| 177 |
-
| `CURRENCY` | Currency |
|
| 178 |
-
| `CURRENCYCODE` | Currencycode |
|
| 179 |
-
| `CURRENCYNAME` | Currencyname |
|
| 180 |
-
| `CURRENCYSYMBOL` | Currencysymbol |
|
| 181 |
-
|
| 182 |
-
</details>
|
| 183 |
-
|
| 184 |
-
<details>
|
| 185 |
-
<summary><strong>Temporal</strong> (2 types)</summary>
|
| 186 |
-
|
| 187 |
-
| Entity | Description |
|
| 188 |
-
|:---|:---|
|
| 189 |
-
| `DATE` | Date |
|
| 190 |
-
| `TIME` | Time |
|
| 191 |
-
|
| 192 |
-
</details>
|
| 193 |
|
| 194 |
## Usage
|
| 195 |
|
| 196 |
-
### Quick Start
|
| 197 |
-
|
| 198 |
```python
|
| 199 |
from transformers import pipeline
|
| 200 |
|
| 201 |
-
|
| 202 |
-
|
| 203 |
-
|
| 204 |
-
|
| 205 |
-
|
| 206 |
-
|
| 207 |
-
العنوان: شارع العليا 42، الرياض 11432.
|
| 208 |
-
"""
|
| 209 |
-
|
| 210 |
-
entities = ner(text)
|
| 211 |
-
for entity in entities:
|
| 212 |
-
print(f"{entity['entity_group']}: {entity['word']} (score: {entity['score']:.3f})")
|
| 213 |
-
```
|
| 214 |
-
|
| 215 |
-
### De-identification Example
|
| 216 |
-
|
| 217 |
-
```python
|
| 218 |
-
def redact_pii(text, entities, placeholder='[REDACTED]'):
|
| 219 |
-
"""Replace detected PII with placeholders."""
|
| 220 |
-
# Sort entities by start position (descending) to preserve offsets
|
| 221 |
-
sorted_entities = sorted(entities, key=lambda x: x['start'], reverse=True)
|
| 222 |
-
redacted = text
|
| 223 |
-
for ent in sorted_entities:
|
| 224 |
-
redacted = redacted[:ent['start']] + f"[{ent['entity_group']}]" + redacted[ent['end']:]
|
| 225 |
-
return redacted
|
| 226 |
-
|
| 227 |
-
# Apply de-identification
|
| 228 |
-
redacted_text = redact_pii(text, entities)
|
| 229 |
-
print(redacted_text)
|
| 230 |
-
```
|
| 231 |
-
|
| 232 |
-
### Batch Processing
|
| 233 |
-
|
| 234 |
-
```python
|
| 235 |
-
from transformers import AutoModelForTokenClassification, AutoTokenizer
|
| 236 |
-
import torch
|
| 237 |
-
|
| 238 |
-
model_name = "OpenMed/OpenMed-PII-Arabic-QwenMed-XLarge-600M-v1"
|
| 239 |
-
model = AutoModelForTokenClassification.from_pretrained(model_name)
|
| 240 |
-
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
| 241 |
|
| 242 |
-
|
| 243 |
-
|
| 244 |
-
"التواصل: khaled.otaibi@email.sa، الهاتف: +966 50 123 4567.",
|
| 245 |
-
]
|
| 246 |
-
|
| 247 |
-
inputs = tokenizer(texts, return_tensors='pt', padding=True, truncation=True)
|
| 248 |
-
with torch.no_grad():
|
| 249 |
-
outputs = model(**inputs)
|
| 250 |
-
predictions = torch.argmax(outputs.logits, dim=-1)
|
| 251 |
```
|
| 252 |
|
| 253 |
-
|
| 254 |
-
|
| 255 |
-
|
| 256 |
-
|
| 257 |
-
This model was trained on a combination of:
|
| 258 |
-
|
| 259 |
-
- **[AI4Privacy PII Masking 200K](https://huggingface.co/datasets/ai4privacy/pii-masking-200k)**: Multilingual base dataset (200K records across 8 languages)
|
| 260 |
-
- **[NVIDIA Nemotron-PII](https://huggingface.co/datasets/nvidia/Nemotron-PII)**: Seed dataset for synthetic data generation
|
| 261 |
-
- **Synthetic Arabic Data**: ~25K high-quality samples generated with locale-specific formatting (National ID format, +966 phones, Arabic names, SAR/ر.س currency)
|
| 262 |
-
|
| 263 |
-
- **Format**: BIO-tagged token classification
|
| 264 |
-
- **Labels**: 76 BIO tags (54 entity types)
|
| 265 |
-
|
| 266 |
-
### Training Configuration
|
| 267 |
|
| 268 |
-
|
| 269 |
-
- **Framework**: Hugging Face Transformers + Trainer API
|
| 270 |
|
| 271 |
-
|
| 272 |
-
|
| 273 |
-
|
| 274 |
-
|
| 275 |
-
- **De-identification**: Automated redaction of PII in Arabic clinical notes, medical records, and documents
|
| 276 |
-
- **Compliance**: Supporting GDPR, and other privacy regulation compliance
|
| 277 |
-
- **Data Preprocessing**: Preparing datasets for research by removing sensitive information
|
| 278 |
-
- **Audit Support**: Identifying PII in document collections
|
| 279 |
-
|
| 280 |
-
### Limitations
|
| 281 |
-
|
| 282 |
-
**Important**: This model is intended as an **assistive tool**, not a replacement for human review.
|
| 283 |
-
|
| 284 |
-
- **False Negatives**: Some PII may not be detected; always verify critical applications
|
| 285 |
-
- **Context Sensitivity**: Performance may vary with domain-specific terminology
|
| 286 |
-
- **Language**: Optimized for Arabic text; may not perform well on other languages
|
| 287 |
-
|
| 288 |
-
## Citation
|
| 289 |
-
|
| 290 |
-
```bibtex
|
| 291 |
-
@misc{openmed-pii-2026,
|
| 292 |
-
title = {OpenMed-PII-Arabic-QwenMed-XLarge-600M-v1: Arabic PII Detection Model},
|
| 293 |
-
author = {OpenMed Science},
|
| 294 |
-
year = {2026},
|
| 295 |
-
publisher = {Hugging Face},
|
| 296 |
-
url = {https://huggingface.co/OpenMed/OpenMed-PII-Arabic-QwenMed-XLarge-600M-v1}
|
| 297 |
-
}
|
| 298 |
-
```
|
| 299 |
|
| 300 |
-
##
|
| 301 |
|
| 302 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
|
|
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- ja
|
| 5 |
base_model: Qwen/Qwen3-Embedding-0.6B
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
library_name: transformers
|
| 7 |
+
pipeline_tag: token-classification
|
| 8 |
+
tags:
|
| 9 |
+
- token-classification
|
| 10 |
+
- ner
|
| 11 |
+
- pii
|
| 12 |
+
- pii-detection
|
| 13 |
+
- de-identification
|
| 14 |
+
- privacy
|
| 15 |
+
- healthcare
|
| 16 |
+
- medical
|
| 17 |
+
- clinical
|
| 18 |
+
- phi
|
| 19 |
+
- pytorch
|
| 20 |
+
- transformers
|
| 21 |
+
- openmed
|
| 22 |
+
- japanese
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
widget:
|
| 24 |
+
- text: 架空の患者、山田太郎さんのメールアドレスは taro.yamada@example.com、電話番号は 090-0000-0000 です。
|
| 25 |
+
example_title: Synthetic clinical text with PII (Japanese)
|
| 26 |
---
|
| 27 |
|
| 28 |
+
# OpenMed-PII-Japanese-QwenMed-XLarge-600M-v1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
|
| 30 |
+
This is an OpenMed token-classification checkpoint intended for Japanese
|
| 31 |
+
(`ja`) personally identifiable information (PII) and protected health
|
| 32 |
+
information (PHI) span detection.
|
|
|
|
|
|
|
| 33 |
|
| 34 |
+
## Model details
|
| 35 |
|
| 36 |
+
- Language scope: Japanese (`ja`)
|
| 37 |
+
- Task: token classification / named entity recognition
|
| 38 |
+
- Base model: [`Qwen/Qwen3-Embedding-0.6B`](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B)
|
| 39 |
+
- Library: Transformers
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
## Usage
|
| 42 |
|
|
|
|
|
|
|
| 43 |
```python
|
| 44 |
from transformers import pipeline
|
| 45 |
|
| 46 |
+
model_id = "OpenMed/OpenMed-PII-Japanese-QwenMed-XLarge-600M-v1"
|
| 47 |
+
detector = pipeline(
|
| 48 |
+
"token-classification",
|
| 49 |
+
model=model_id,
|
| 50 |
+
aggregation_strategy="simple",
|
| 51 |
+
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
|
| 53 |
+
text = '架空の患者、山田太郎さんのメールアドレスは taro.yamada@example.com、電話番号は 090-0000-0000 です。'
|
| 54 |
+
print(detector(text))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
```
|
| 56 |
|
| 57 |
+
The checkpoint's configured `id2label` mapping is authoritative for the
|
| 58 |
+
available entity labels. Preserve returned character offsets when applying
|
| 59 |
+
redaction or replacement.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
|
| 61 |
+
## Evaluation status
|
|
|
|
| 62 |
|
| 63 |
+
No verified Japanese evaluation artifact was available during this metadata
|
| 64 |
+
repair, so this card intentionally reports no language-specific scores.
|
| 65 |
+
Evaluate direct-identifier recall, false negatives, span boundaries, and
|
| 66 |
+
domain shift on representative data before deployment.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
|
| 68 |
+
## Limitations and safety
|
| 69 |
|
| 70 |
+
This model can miss identifiers or over-redact clinically useful context. It
|
| 71 |
+
is not an anonymization guarantee, a compliance determination, or a medical
|
| 72 |
+
device. Use defense in depth and human review for high-sensitivity workflows.
|
| 73 |
+
Do not include real patient information in public examples, logs, or issue
|
| 74 |
+
reports.
|