Add Indonesian NER spaCy model with comprehensive documentation

- Add complete spaCy model files with NER pipeline for Indonesian text
- Include detailed performance metrics (F1: 0.9856, Precision: 0.9846, Recall: 0.9865)
- Add comprehensive README with training configuration and architecture details
- Support 19 entity types with per-entity performance breakdown
- Configure LFS tracking for large model files
- Include evaluation tools and usage instructions

Author: Asep Muhamad <[email protected]>

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+vocab/vectors filter=lfs diff=lfs merge=lfs -text
+ner/model filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,150 @@
 ---
 license: gpl-2.0
+language: id
+tags:
+- spacy
+- ner
+- token-classification
+- indonesian
+library_name: spacy
 ---
+
+# Indonesian NER spaCy Model
+
+This model is a Named Entity Recognition (NER) model for Indonesian language built with spaCy.
+
+## Model Details
+
+- **Language**: Indonesian (`id`)
+- **Pipeline**: `ner`
+- **spaCy Version**: `>=3.8.7,<3.9.0`
+- **Model Architecture**: Transition-based parser with HashEmbedCNN tok2vec
+
+## Supported Entity Types
+
+The model recognizes the following entity types:
+
+- `CARDINAL` - Cardinal numbers
+- `DATE` - Date expressions
+- `EVENT` - Events
+- `FACILITY` - Facilities
+- `GPE` - Geopolitical entities
+- `LANGUAGE` - Languages
+- `LAW` - Legal documents
+- `LOCATION` - Locations
+- `MISC` - Miscellaneous
+- `MONEY` - Monetary values
+- `NORP` - Nationalities or religious/political groups
+- `ORDINAL` - Ordinal numbers
+- `ORGANIZATION` - Organizations
+- `PERCENT` - Percentages
+- `PERSON` - People
+- `PRODUCT` - Products
+- `QUANTITY` - Quantities
+- `TIME` - Time expressions
+- `TITLE` - Titles
+
+## Usage
+
+```python
+import spacy
+
+# Load the model
+nlp = spacy.load("asmud/ner-spacy-indonesian")
+
+# Process text
+doc = nlp("Presiden Joko Widodo mengunjungi Jakarta pada tanggal 17 Agustus 2024.")
+
+# Extract entities
+for ent in doc.ents:
+    print(f"{ent.text} -> {ent.label_}")
+```
+
+## Installation
+
+```bash
+pip install https://huggingface.co/asmud/ner-spacy-indonesian/resolve/main/ner-spacy-indonesian-any-py3-none-any.whl
+```
+
+Or use with spaCy:
+
+```python
+import spacy
+nlp = spacy.load("asmud/ner-spacy-indonesian")
+```
+
+## Model Architecture
+
+- **tok2vec**: HashEmbedCNN with 96-dimensional embeddings, depth 4, embed size 2000
+- **ner**: Transition-based parser with 64 hidden units, maxout pieces 2
+- **Training**: 100 iterations with dropout 0.5, compounding batch sizes (4-32)
+- **Optimizer**: Adam (lr=0.001, L2=0.01, grad_clip=1.0)
+
+## Training Configuration
+
+### Training Data Format
+The model was trained on data with custom XML-like tags:
+```
+Presiden <PERSON>Joko Widodo</PERSON> mengunjungi <GPE>Jakarta</GPE> pada <DATE>17 Agustus 2024</DATE>.
+```
+
+### Training Parameters
+- **Iterations**: 100 training iterations
+- **Dropout**: 0.5 during training
+- **Batch Size**: Compounding from 4 to 32 examples
+- **Text Preprocessing**: Lowercased input text
+- **Data Shuffling**: Random shuffling each iteration
+
+### Architecture Details
+- **Embedding Width**: 96 dimensions
+- **Hidden Width**: 64 units
+- **Embed Size**: 2000 features
+- **Window Size**: 1
+- **Maxout Pieces**: 3 (tok2vec), 2 (parser)
+- **Subword Features**: Enabled
+
+## Model Evaluation
+
+### Performance Metrics
+The model was evaluated on 2,987 examples from the training data with the following results:
+
+#### Overall Performance
+- **Precision**: 0.9846
+- **Recall**: 0.9865
+- **F1-score**: 0.9856
+
+#### Per-Entity Performance
+| Entity | Precision | Recall | F1-score |
+|--------|-----------|--------|----------|
+| PRODUCT | 1.0000 | 1.0000 | 1.0000 |
+| LOCATION | 1.0000 | 1.0000 | 1.0000 |
+| LANGUAGE | 1.0000 | 1.0000 | 1.0000 |
+| EVENT | 0.9962 | 1.0000 | 0.9981 |
+| MISC | 0.9973 | 0.9960 | 0.9966 |
+| FACILITY | 0.9923 | 1.0000 | 0.9961 |
+| LAW | 1.0000 | 0.9919 | 0.9959 |
+| TITLE | 0.9947 | 0.9947 | 0.9947 |
+| GPE | 1.0000 | 0.9886 | 0.9943 |
+| NORP | 0.9872 | 1.0000 | 0.9935 |
+| PERSON | 0.9935 | 0.9935 | 0.9935 |
+| DATE | 0.9926 | 0.9830 | 0.9878 |
+| ORDINAL | 0.9750 | 1.0000 | 0.9873 |
+| MONEY | 0.9683 | 0.9946 | 0.9812 |
+| ORGANIZATION | 0.9457 | 0.9905 | 0.9676 |
+| TIME | 0.9476 | 0.9819 | 0.9645 |
+| QUANTITY | 0.9874 | 0.9291 | 0.9574 |
+| PERCENT | 0.8600 | 1.0000 | 0.9247 |
+| CARDINAL | 0.9620 | 0.8736 | 0.9157 |
+
+### Evaluation Features
+You can reproduce these metrics using the included analyzer script:
+
+```bash
+streamlit run spacy_model_analyzer.py
+```
+
+The analyzer provides:
+- **Interactive Analysis**: Real-time entity recognition testing
+- **Detailed Metrics**: Precision, recall, and F1-score calculations
+- **Text Alignment**: Automatic handling of entity boundary alignment
+- **Visualization**: Entity highlighting and analysis tools

config.cfg

@@ -0,0 +1,130 @@
+[paths]
+train = null
+dev = null
+vectors = null
+init_tok2vec = null
+
+[system]
+seed = 0
+gpu_allocator = null
+
+[nlp]
+lang = "id"
+pipeline = ["ner"]
+disabled = []
+before_creation = null
+after_creation = null
+after_pipeline_creation = null
+batch_size = 1000
+tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
+vectors = {"@vectors":"spacy.Vectors.v1"}
+
+[components]
+
+[components.ner]
+factory = "ner"
+incorrect_spans_key = null
+moves = null
+scorer = {"@scorers":"spacy.ner_scorer.v1"}
+update_with_oracle_cut_size = 100
+
+[components.ner.model]
+@architectures = "spacy.TransitionBasedParser.v2"
+state_type = "ner"
+extra_state_tokens = false
+hidden_width = 64
+maxout_pieces = 2
+use_upper = true
+nO = null
+
+[components.ner.model.tok2vec]
+@architectures = "spacy.HashEmbedCNN.v2"
+pretrained_vectors = null
+width = 96
+depth = 4
+embed_size = 2000
+window_size = 1
+maxout_pieces = 3
+subword_features = true
+
+[corpora]
+
+[corpora.dev]
+@readers = "spacy.Corpus.v1"
+path = ${paths.dev}
+gold_preproc = false
+max_length = 0
+limit = 0
+augmenter = null
+
+[corpora.train]
+@readers = "spacy.Corpus.v1"
+path = ${paths.train}
+gold_preproc = false
+max_length = 0
+limit = 0
+augmenter = null
+
+[training]
+seed = ${system.seed}
+gpu_allocator = ${system.gpu_allocator}
+dropout = 0.1
+accumulate_gradient = 1
+patience = 1600
+max_epochs = 0
+max_steps = 20000
+eval_frequency = 200
+frozen_components = []
+annotating_components = []
+dev_corpus = "corpora.dev"
+train_corpus = "corpora.train"
+before_to_disk = null
+before_update = null
+
+[training.batcher]
+@batchers = "spacy.batch_by_words.v1"
+discard_oversize = false
+tolerance = 0.2
+get_length = null
+
+[training.batcher.size]
+@schedules = "compounding.v1"
+start = 100
+stop = 1000
+compound = 1.001
+t = 0.0
+
+[training.logger]
+@loggers = "spacy.ConsoleLogger.v1"
+progress_bar = false
+
+[training.optimizer]
+@optimizers = "Adam.v1"
+beta1 = 0.9
+beta2 = 0.999
+L2_is_weight_decay = true
+L2 = 0.01
+grad_clip = 1.0
+use_averages = false
+eps = 0.00000001
+learn_rate = 0.001
+
+[training.score_weights]
+ents_f = 1.0
+ents_p = 0.0
+ents_r = 0.0
+ents_per_type = null
+
+[pretraining]
+
+[initialize]
+vectors = ${paths.vectors}
+init_tok2vec = ${paths.init_tok2vec}
+vocab_data = null
+lookups = null
+before_init = null
+after_init = null
+
+[initialize.components]
+
+[initialize.tokenizer]

meta.json

@@ -0,0 +1,51 @@
+{
+  "lang":"id",
+  "name":"pipeline",
+  "version":"0.0.0",
+  "spacy_version":">=3.8.7,<3.9.0",
+  "description":"",
+  "author":"",
+  "email":"",
+  "url":"",
+  "license":"",
+  "spacy_git_version":"4b65aa7",
+  "vectors":{
+    "width":0,
+    "vectors":0,
+    "keys":0,
+    "name":null,
+    "mode":"default"
+  },
+  "labels":{
+    "ner":[
+      "CARDINAL",
+      "DATE",
+      "EVENT",
+      "FACILITY",
+      "GPE",
+      "LANGUAGE",
+      "LAW",
+      "LOCATION",
+      "MISC",
+      "MONEY",
+      "NORP",
+      "ORDINAL",
+      "ORGANIZATION",
+      "PERCENT",
+      "PERSON",
+      "PRODUCT",
+      "QUANTITY",
+      "TIME",
+      "TITLE"
+    ]
+  },
+  "pipeline":[
+    "ner"
+  ],
+  "components":[
+    "ner"
+  ],
+  "disabled":[
+
+  ]
+}

ner/cfg

@@ -0,0 +1,13 @@
+{
+  "moves":null,
+  "update_with_oracle_cut_size":100,
+  "multitasks":[
+
+  ],
+  "min_action_freq":1,
+  "learn_tokens":false,
+  "beam_width":1,
+  "beam_density":0.0,
+  "beam_update_prob":0.0,
+  "incorrect_spans_key":null
+}

ner/model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f6fd3968fdd127d227a20c184745f81f86fa63c1af9f8de767d2c2db70e83a73
+size 3851641

ner/moves

@@ -0,0 +1 @@
+��moves��{"0":{},"1":{"MISC":-1,"MONEY":-2,"LAW":-3,"DATE":-4,"FACILITY":-5,"PRODUCT":-6,"EVENT":-7,"ORDINAL":-8,"GPE":-9,"PERCENT":-10,"LOCATION":-11,"TITLE":-12,"ORGANIZATION":-13,"TIME":-14,"QUANTITY":-15,"NORP":-16,"LANGUAGE":-17,"CARDINAL":-18,"PERSON":-19},"2":{"MISC":-1,"MONEY":-2,"LAW":-3,"DATE":-4,"FACILITY":-5,"PRODUCT":-6,"EVENT":-7,"ORDINAL":-8,"GPE":-9,"PERCENT":-10,"LOCATION":-11,"TITLE":-12,"ORGANIZATION":-13,"TIME":-14,"QUANTITY":-15,"NORP":-16,"LANGUAGE":-17,"CARDINAL":-18,"PERSON":-19},"3":{"MISC":-1,"MONEY":-2,"LAW":-3,"DATE":-4,"FACILITY":-5,"PRODUCT":-6,"EVENT":-7,"ORDINAL":-8,"GPE":-9,"PERCENT":-10,"LOCATION":-11,"TITLE":-12,"ORGANIZATION":-13,"TIME":-14,"QUANTITY":-15,"NORP":-16,"LANGUAGE":-17,"CARDINAL":-18,"PERSON":-19},"4":{"":1,"MISC":-1,"MONEY":-2,"LAW":-3,"DATE":-4,"FACILITY":-5,"PRODUCT":-6,"EVENT":-7,"ORDINAL":-8,"GPE":-9,"PERCENT":-10,"LOCATION":-11,"TITLE":-12,"ORGANIZATION":-13,"TIME":-14,"QUANTITY":-15,"NORP":-16,"LANGUAGE":-17,"CARDINAL":-18,"PERSON":-19},"5":{"":1}}�cfg��neg_key�

vocab/key2row

@@ -0,0 +1 @@
+�

vocab/lookups.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:76be8b528d0075f7aae98d6fa57a6d3c83ae480a8469e668d7b0af968995ac71
+size 1

vocab/vectors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:14772b683e726436d5948ad3fff2b43d036ef2ebbe3458aafed6004e05a40706
+size 128

vocab/vectors.cfg

@@ -0,0 +1,3 @@
+{
+  "mode":"default"
+}