---
language: en
license: mit
tags:
- fasttext
- tajik
- word-embeddings
- nlp
---

# Tajik FastText Word Embedding Model

This repository contains a pretrained **FastText** model for the **Tajik language**, trained on a large corpus of Tajik texts. The model supports **subword information**, allowing it to generate embeddings even for rare or unseen (OOV) words.

The model is suitable for use in various NLP tasks such as:
- Semantic analysis
- Text classification
- Machine translation
- Synonym detection and thesaurus building
- Enhancing other models through embedding initialization

Licensed under the [MIT License](LICENSE), which allows free usage in both research and commercial applications.

---

## 📊 Model Overview

| Parameter         | Value                      |
|------------------|----------------------------|
| Model Type        | FastText (with subwords)   |
| Vector Size       | 300                        |
| Vocabulary Size   | 145,232                    |
| OOV Support       | Yes                        |
| Context Window    | 5                          |
| Min Word Count    | ≥ 5                        |

---

## 📚 Training Corpus

### Books (Total: 99)
- Programming: 6
- History: 4
- Religion: 12
- Scientific: 3
- Children's literature: 6
- Prose: 19
- Poetry: 21
- Textbooks: 28

### Articles (Total: 134,497)
- Asia-Plus: 20,471
- Khovar: 21,557
- Ovozi Tojik: 7,495
- Farazh: 4,679
- Wikipedia: 80,295

### Total Corpus Statistics
- **Documents**: 134,596
- **Tokens**: 33,535,383
- **Unique Lemmas**: 649,308

---

## 🧪 Model Comparison with Meta FastText

We evaluated our model against Meta’s pretrained FastText using semantic similarity and Spearman correlation:

| Model             | Spearman Correlation | OOV Support |
|------------------|----------------------|-------------|
| FastText (Meta)  | **0.703**            | Yes         |
| **FastText (ours)** | **0.622**        | **Yes**     |

While Meta FastText achieves better overall performance, our model demonstrates strong results on Tajik-specific morphology and semantics.

---

## 🔍 Example Similar Words

| Word      | Nearest Neighbors (FastText) |
|-----------|-------------------------------|
| кӯдак     | кӯдаку(0.82), хурдкӯдак(0.81), кӯдакам(0.81), кӯдакат(0.81), кӯдаке(0.81) |
| муаллим   | муаллиме(0.90), муаллимат(0.89), муаллимин(0.89), муаллиму(0.88), муаллима(0.88) |
| об        | оби(0.79), обро(0.74), обмӯрии(0.70), обшустаи(0.68), обшуста(0.66) |
| мард      | марда(0.87), мардхӯ(0.85), мардвор(0.85), мардро(0.83), зан(0.82) |
| деҳа      | деҳайи(0.83), деҳаю(0.80), деҳавз(0.78), деҳакӣ(0.76), деҳодеҳ(0.74) |
| китоб     | китобӣ(0.84), китобгуна(0.83), китобча(0.81), китобсӯзӣ(0.81), китобро(0.81) |
| меҳмон    | меҳмонӣ(0.86), меҳмоншо(0.85), меҳмонат(0.83), меҳмонҳона(0.82), меҳмони(0.82) |
| шаҳр      | шаҳрӯ(0.82), шаҳрча(0.80), бушаҳр(0.79), шаҳрат(0.79), навшаҳр(0.79) |
| падар     | падаршӯ(0.89), падарӣ(0.84), падаршӯву(0.84), падаре(0.84), падаршон(0.83) |
| модар     | модаршӯ(0.86), модаршӯяш(0.83), модару(0.81), модаре(0.81), модарвор(0.80) |

---

## 🧩 Handling OOV (Out-of-Vocabulary) Words

FastText supports generating vectors for unknown words via subword units (n-grams). Here are some examples:

| Unknown Word | Closest Matches (FastText) |
|--------------|----------------------------|
| кӯдакона     | кӯдаконаи(0.82), кӯдаконат(0.81), кӯдаконае(0.81) |
| меҳмонамон   | меҳмон(0.77), меҳмонҳо(0.77), меҳмонам(0.76) |
| муаллимон    | муаллимони(0.89), муаллимоне(0.88), муаллимону(0.83) |
| деҳоти       | дарҷамоати(0.79), чамоати(0.74), ҷамоати(0.81) |
| саводнок     | саводнокӣ(0.88), саводнокиву(0.85), саводнокии(0.84) |

---

## 📌 Features for Tajik Language

Our model performs well on:
- **Semantic similarity**: e.g., "мард" ↔ "зан", "китоб" ↔ "китобгуна"
- **Morphological variants**: e.g., "кӯдак" → "кӯдаку", "кӯдаки"
- **Rare/compound words**: thanks to subword representations like "саводнок", "деҳоти"

---

## 💡 Usage Example

```python
from gensim.models import FastText

model = FastText.load("tajik_fasttext.model")
vector = model.wv["падар"]  # Get vector for a word
similar_words = model.wv.most_similar("модар")  # Find similar words
```

---

## 🗂️ Files Included

| File               | Description                                  |
|--------------------|----------------------------------------------|
| `tajik_fasttext.model` | Gensim FastText model file                 |
| `*.npy` files         | Supporting NumPy arrays for vectors        |

---

## 📚 Citation

If you use this model, please cite:

```bibtex
@misc{ArabovMK_Tajik_FastText,
  author = {ArabovMK},
  title = {Tajik FastText Word Embeddings},
  year = 2025,
  publisher = {Hugging Face},
  url = {https://huggingface.co/ArabovMK/tajik-fasttext-model}
}
```

*Last updated: 2025-05-10*