Upload all models and assets for cu (latest)

.gitattributes

@@ -39,3 +39,4 @@ visualizations/position_encoding_comparison.png filter=lfs diff=lfs merge=lfs -t
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: cu
-language_name: CU
+language_name: Church Slavic
 language_family: slavic_historical
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-slavic_historical
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,20 +33,20 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.945
+    value: 4.940
   - name: best_isotropy
     type: isotropy
-    value: 0.2996
+    value: 0.2434
   - name: vocabulary_size
     type: vocab
     value: 0
 generated: 2026-01-03
 ---
 
-# CU - Wikilangs Models
+# Church Slavic - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **CU** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Church Slavic** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -60,7 +70,7 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
 - [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
@@ -80,43 +90,43 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.879x | 3.88 | 0.1306% | 107,930 |
-| **16k** | 4.369x | 4.37 | 0.1472% | 95,813 |
-| **32k** | 4.945x 🏆 | 4.95 | 0.1666% | 84,659 |
+| **8k** | 3.877x | 3.88 | 0.1314% | 107,273 |
+| **16k** | 4.367x | 4.37 | 0.1480% | 95,246 |
+| **32k** | 4.940x 🏆 | 4.94 | 0.1675% | 84,200 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `thumb Ꚛ (имѧ Ꙋкрестъ) словѣньскаѥго ѩꙁꙑка боукꙑ ѥстъ. ѩꙁꙑка боукъви`
+**Sample 1:** `Лидьскъ повѣтъ · Бѣла Роусь Лидьскъ повѣтъ · Рѡсїиска їмпєрїꙗ`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁thumb ▁ ꚛ ▁( имѧ ▁ꙋ к ре стъ ) ... (+7 more)` | 17 |
-| 16k | `▁thumb ▁ ꚛ ▁( имѧ ▁ꙋ кре стъ ) ▁словѣньскаѥго ... (+6 more)` | 16 |
-| 32k | `▁thumb ▁ ꚛ ▁( имѧ ▁ꙋкрестъ ) ▁словѣньскаѥго ▁ѩꙁꙑка ▁боукꙑ ... (+4 more)` | 14 |
+| 8k | `▁лидьскъ ▁повѣтъ ▁· ▁бѣла ▁роусь ▁лидьскъ ▁повѣтъ ▁· ▁рѡсїиска ▁їмпєрїꙗ` | 10 |
+| 16k | `▁лидьскъ ▁повѣтъ ▁· ▁бѣла ▁роусь ▁лидьскъ ▁повѣтъ ▁· ▁рѡсїиска ▁їмпєрїꙗ` | 10 |
+| 32k | `▁лидьскъ ▁повѣтъ ▁· ▁бѣла ▁роусь ▁лидьскъ ▁повѣтъ ▁· ▁рѡсїиска ▁їмпєрїꙗ` | 10 |
 
-**Sample 2:** `Могилєвъ и · · градъ Бѣлꙑ Роуси ѥстъ ⁙ Людии обитаѥтъ 371 318 ⁙ Помѣновєнъ жє ꙁа...`
+**Sample 2:** `Оꙁаскоу и · юга Санъ Паоулоу браꙁїльскъ градъ и обьщина ѥстъ ⁙ Людии 718.646 оби...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁могилєвъ ▁и ▁· ▁· ▁градъ ▁бѣлꙑ ▁роуси ▁ѥстъ ▁⁙ ▁людии ... (+31 more)` | 41 |
-| 16k | `▁могилєвъ ▁и ▁· ▁· ▁градъ ▁бѣлꙑ ▁роуси ▁ѥстъ ▁⁙ ▁людии ... (+28 more)` | 38 |
-| 32k | `▁могилєвъ ▁и ▁· ▁· ▁градъ ▁бѣлꙑ ▁роуси ▁ѥстъ ▁⁙ ▁людии ... (+27 more)` | 37 |
+| 8k | `▁о ꙁа скоу ▁и ▁· ▁ю га ▁санъ ▁паоу лоу ... (+24 more)` | 34 |
+| 16k | `▁о ꙁа скоу ▁и ▁· ▁ю га ▁санъ ▁паоулоу ▁браꙁїл ... (+23 more)` | 33 |
+| 32k | `▁оꙁаскоу ▁и ▁· ▁юга ▁санъ ▁паоулоу ▁браꙁїльскъ ▁градъ ▁и ▁обьщина ... (+19 more)` | 29 |
 
-**Sample 3:** `thumb Ѱ (имѧ ыпсьлон) словѣньскаѥго ѩꙁꙑка боукꙑ ѥстъ ѩꙁꙑка боукъви аꙁъбоукꙑ боук...`
+**Sample 3:** `Октадєканъ и инако н-октадєканъ ѫглѥводородьно вєщьство алканъ рѧдоу ѥстъ ⁙ Ѥгож...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁thumb ▁ѱ ▁( имѧ ▁ы п сь лон ) ▁словѣньскаѥго ... (+7 more)` | 17 |
-| 16k | `▁thumb ▁ѱ ▁( имѧ ▁ы п сь лон ) ▁словѣньскаѥго ... (+7 more)` | 17 |
-| 32k | `▁thumb ▁ѱ ▁( имѧ ▁ыпсьлон ) ▁словѣньскаѥго ▁ѩꙁꙑка ▁боукꙑ ▁ѥстъ ... (+4 more)` | 14 |
+| 8k | `▁ок тадєканъ ▁и ▁инако ▁н - ок тадєканъ ▁ѫглѥводородьно ▁вєщьство ... (+19 more)` | 29 |
+| 16k | `▁октадєканъ ▁и ▁инако ▁н - ок тадєканъ ▁ѫглѥводородьно ▁вєщьство ▁алканъ ... (+17 more)` | 27 |
+| 32k | `▁октадєканъ ▁и ▁инако ▁н - октадєканъ ▁ѫглѥводородьно ▁вєщьство ▁алканъ ▁рѧдоу ... (+16 more)` | 26 |
 
 
 ### Key Findings
 
-- **Best Compression:** 32k achieves 4.945x compression
-- **Lowest UNK Rate:** 8k with 0.1306% unknown tokens
+- **Best Compression:** 32k achieves 4.940x compression
+- **Lowest UNK Rate:** 8k with 0.1314% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -133,12 +143,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
 |--------|---------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | Word | 803 | 9.65 | 1,418 | 38.7% | 88.8% |
-| **2-gram** | Subword | 451 🏆 | 8.82 | 2,626 | 56.3% | 95.5% |
-| **3-gram** | Word | 974 | 9.93 | 1,743 | 35.2% | 82.0% |
-| **3-gram** | Subword | 2,632 | 11.36 | 12,321 | 25.6% | 67.4% |
-| **4-gram** | Word | 1,602 | 10.65 | 2,970 | 29.2% | 66.7% |
-| **4-gram** | Subword | 8,242 | 13.01 | 33,307 | 16.1% | 45.2% |
+| **2-gram** | Word | 802 | 9.65 | 1,417 | 38.7% | 88.9% |
+| **2-gram** | Subword | 451 🏆 | 8.82 | 2,622 | 56.3% | 95.5% |
+| **3-gram** | Word | 965 | 9.91 | 1,734 | 35.4% | 82.3% |
+| **3-gram** | Subword | 2,629 | 11.36 | 12,286 | 25.7% | 67.4% |
+| **4-gram** | Word | 1,583 | 10.63 | 2,960 | 29.4% | 67.1% |
+| **4-gram** | Subword | 8,218 | 13.00 | 33,187 | 16.1% | 45.2% |
+| **5-gram** | Word | 1,176 | 10.20 | 2,224 | 32.9% | 74.0% |
+| **5-gram** | Subword | 14,289 | 13.80 | 46,031 | 12.7% | 35.8% |
 
 ### Top 5 N-grams by Size
 
@@ -146,7 +158,7 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `ꙁьри такождє` | 429 |
+| 1 | `ꙁьри такождє` | 432 |
 | 2 | `людии обитаѥтъ` | 260 |
 | 3 | `ѥстъ людии` | 234 |
 | 4 | `градъ ѥстъ` | 230 |
@@ -157,57 +169,77 @@ Below are sample sentences tokenized with each vocabulary size:
 | Rank | N-gram | Count |
 |------|--------|-------|
 | 1 | `ѥстъ людии обитаѥтъ` | 181 |
-| 2 | `въ дрьжавѣ бѣла` | 120 |
-| 3 | `дрьжавѣ бѣла роусь` | 120 |
+| 2 | `дрьжавѣ бѣла роусь` | 120 |
+| 3 | `въ дрьжавѣ бѣла` | 120 |
 | 4 | `градъ ѥстъ людии` | 115 |
-| 5 | `роусь сѣи оудѣлъ` | 114 |
+| 5 | `бѣла роусь сѣи` | 114 |
 
 **4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
 | 1 | `въ дрьжавѣ бѣла роусь` | 120 |
-| 2 | `ꙁємьскъ оудѣлъ въ дрьжавѣ` | 114 |
+| 2 | `дрьжавѣ бѣла роусь сѣи` | 114 |
 | 3 | `оудѣлъ въ дрьжавѣ бѣла` | 114 |
-| 4 | `дрьжавѣ бѣла роусь сѣи` | 114 |
-| 5 | `ѥстъ ꙁємьскъ оудѣлъ въ` | 114 |
+| 4 | `ꙁємьскъ оудѣлъ въ дрьжавѣ` | 114 |
+| 5 | `бѣла роусь сѣи оудѣлъ` | 114 |
+
+**5-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `роусь сѣи оудѣлъ бѣ члѣнъ` | 114 |
+| 2 | `ꙁємьскъ оудѣлъ въ дрьжавѣ бѣла` | 114 |
+| 3 | `оудѣлъ въ дрьжавѣ бѣла роусь` | 114 |
+| 4 | `бѣла роусь сѣи оудѣлъ бѣ` | 114 |
+| 5 | `дрьжавѣ бѣла роусь сѣи оудѣлъ` | 114 |
 
 **2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `ъ _` | 17,731 |
-| 2 | `и _` | 9,203 |
-| 3 | `а _` | 8,612 |
-| 4 | `с т` | 8,393 |
-| 5 | `_ с` | 6,604 |
+| 1 | `ъ _` | 17,697 |
+| 2 | `и _` | 9,192 |
+| 3 | `а _` | 8,589 |
+| 4 | `с т` | 8,369 |
+| 5 | `_ с` | 6,568 |
 
 **3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `т ъ _` | 5,941 |
-| 2 | `_ · _` | 4,423 |
-| 3 | `ь с к` | 3,904 |
-| 4 | `_ ⁙ _` | 3,096 |
-| 5 | `с т ъ` | 3,041 |
+| 1 | `т ъ _` | 5,939 |
+| 2 | `_ · _` | 4,413 |
+| 3 | `ь с к` | 3,883 |
+| 4 | `_ ⁙ _` | 3,094 |
+| 5 | `с т ъ` | 3,038 |
 
 **4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `_ ѥ с т` | 2,898 |
-| 2 | `с т ъ _` | 2,880 |
-| 3 | `ѥ с т ъ` | 2,700 |
-| 4 | `ъ _ ⁙ _` | 1,900 |
-| 5 | `т ъ _ ⁙` | 1,811 |
+| 1 | `_ ѥ с т` | 2,895 |
+| 2 | `с т ъ _` | 2,876 |
+| 3 | `ѥ с т ъ` | 2,698 |
+| 4 | `ъ _ ⁙ _` | 1,902 |
+| 5 | `т ъ _ ⁙` | 1,813 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ ѥ с т ъ` | 2,695 |
+| 2 | `ѥ с т ъ _` | 2,559 |
+| 3 | `т ъ _ ⁙ _` | 1,796 |
+| 4 | `_ г р а д` | 1,425 |
+| 5 | `с т ъ _ ⁙` | 1,340 |
 
 
 ### Key Findings
 
 - **Best Perplexity:** 2-gram (subword) with 451
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~45% of corpus
+- **Coverage:** Top-1000 patterns cover ~36% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -223,14 +255,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
 |---------|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | Word | 0.4875 | 1.402 | 2.62 | 18,795 | 51.2% |
-| **1** | Subword | 0.9912 | 1.988 | 7.09 | 1,078 | 0.9% |
-| **2** | Word | 0.1229 | 1.089 | 1.22 | 48,721 | 87.7% |
-| **2** | Subword | 0.8197 | 1.765 | 4.19 | 7,639 | 18.0% |
-| **3** | Word | 0.0442 | 1.031 | 1.07 | 58,656 | 95.6% |
-| **3** | Subword | 0.5526 | 1.467 | 2.43 | 31,947 | 44.7% |
-| **4** | Word | 0.0206 🏆 | 1.014 | 1.03 | 61,545 | 97.9% |
-| **4** | Subword | 0.3393 | 1.265 | 1.70 | 77,657 | 66.1% |
+| **1** | Word | 0.4863 | 1.401 | 2.62 | 18,746 | 51.4% |
+| **1** | Subword | 0.9940 | 1.992 | 7.09 | 1,077 | 0.6% |
+| **2** | Word | 0.1229 | 1.089 | 1.22 | 48,473 | 87.7% |
+| **2** | Subword | 0.8201 | 1.766 | 4.18 | 7,633 | 18.0% |
+| **3** | Word | 0.0444 | 1.031 | 1.07 | 58,365 | 95.6% |
+| **3** | Subword | 0.5514 | 1.466 | 2.43 | 31,900 | 44.9% |
+| **4** | Word | 0.0207 🏆 | 1.014 | 1.03 | 61,255 | 97.9% |
+| **4** | Subword | 0.3387 | 1.265 | 1.70 | 77,420 | 66.1% |
 
 ### Generated Text Samples (Word-based)
 
@@ -238,27 +270,27 @@ Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `и словѣньскъ ѩꙁꙑкъ ѥстъ людии обитаѥтъ стольнъ градъ ѥстъ ѥгожє потомъць тєодєнъ ꙗко идєжє kb постоꙗ...`
-2. `ѥстъ додєканъ ѫглѥводородьно вєщьство алканъ рѧдоу ѥстъ ꙗко нарочито поѩтиѥ паоуло коєлио пїитъ браꙁ...`
-3. `лѣта нарєчєнъ съ тꙑлоу жєнꙑ ѳєологїѩ вївлїи въ дрьжавѣ бѣла роусь сѣи оудѣлъ въ дрьжавѣ бѣла`
+1. `и ꙁападьнꙑ дъвинꙑ роуси пьсаниꙗ алєѯандра данїиловища свѣтьлѣиша кънѧꙃа владєнию бѣ съ словѣньскомь ...`
+2. `ѥстъ стольнъ градъ ѥстъ нѣмьцкомь єпископомь албєртомь а нꙑнѣ жє носьнꙑи приꙁвѫкъ нє ꙁнаашє ѥдьнъ ис`
+3. `лѣта їмпєратѡръ ѥстъ пєроунъ сварогъ ѩꙁꙑчьство`
 
 **Context Size 2:**
 
-1. `ꙁьри такождє владимѣръ мєждоусѣтии гради гради въ асии аꙁєрбаичаноѵ`
-2. `людии обитаѥтъ масачоусєтсѣ 7 лєѡдръ обитаѥтъ таджикистана дрьжавьнъ ѩꙁꙑкъ соуми ѥстъ симъ ѩꙁꙑкомъ 9...`
-3. `ѥстъ людии обитаѥтъ лѣта 788 лѣто 168 17 64 320 0 10 23 ꙁапражиѥиванофранковьска 13 9 13`
+1. `ꙁьри такождє обитѣльско напьсаниѥ владиславъ їѡаннъ асєн҄ь а҃ и блъгарїꙗ цѣсарь бѣ їѡанна асєнꙗ а҃ с...`
+2. `людии обитаѥтъ 6 9 лєѡ́дръ їсторїꙗ лѣта по нѣмьць ѥдьнѥниꙗ бєрлинъ пакꙑ сталъ ѥстъ ꙁьри такъждє брюѯ...`
+3. `ѥстъ людии 2 лєѡдръ обитаѥтъ пакистана дрьжавьнъ ѩꙁꙑкъ тѷрчьскъ ѥстъ їсторїꙗ дѣлꙗ охранꙑ съдравиꙗ лѣ...`
 
 **Context Size 3:**
 
-1. `ѥстъ людии обитаѥтъ 700 тꙑсѫщь основанъ ѥстъ лѣта нарєчєнъ градъ съ лѣта гєѡргїꙗ жє мьнитъ лꙑхнꙑ ꙗко...`
-2. `въ дрьжавѣ бѣла роусь сѣи оудѣлъ бѣ члѣнъ ѡбласти рѣкома мѣньска ѡбласть повѣтъ имаѥтъ оурѧдъ рѣкомъ...`
-3. `дрьжавѣ бѣла роусь сѣи оудѣлъ бѣ члѣнъ ѡбласти рѣкома бєрєстєиска ѡбласть повѣтъ имаѥтъ оурѧдъ рѣком...`
+1. `ѥстъ людии обитаѥтъ 398 и иꙁъ ихъжє мѫжь 175 и жєнъ 223 наибол҄ии числомь народъ роусьсци ѥстъ 99`
+2. `въ дрьжавѣ бѣла роусь сѣи оудѣлъ бѣ члѣнъ ѡбласти рѣкома могилєвьска ѡбласть повѣтъ имаѥтъ оурѧдъ рѣ...`
+3. `дрьжавѣ бѣла роусь сѣи оудѣлъ бѣ члѣнъ градоу витьбьскъ въ ѡбласти рѣкома мѣньска ѡбласть повѣтъ има...`
 
 **Context Size 4:**
 
-1. `въ дрьжавѣ бѣла роусь сѣи оудѣлъ бѣ члѣнъ ѡбласти рѣкома мѣньска ѡбласть конѣць иматъ оурѧдъ рѣкомъ ...`
-2. `ѥстъ ꙁємьскъ оудѣлъ въ дрьжавѣ бѣла роусь сѣи оудѣлъ бѣ члѣнъ градоу витьбьскъ въ ѡбласти рѣкома вит...`
-3. `ꙁємьскъ оудѣлъ въ дрьжавѣ бѣла роусь сѣи оудѣлъ бѣ члѣнъ ѡбласти рѣкома бєрєстєиска ѡбласть повѣтъ и...`
+1. `въ дрьжавѣ бѣла роусь сѣи оудѣлъ бѣ члѣнъ ѡбласти рѣкома бєрєстєиска ѡбласть повѣтъ имаѥтъ оурѧдъ рѣ...`
+2. `роусь сѣи оудѣлъ бѣ члѣнъ ѡбласти рѣкома мѣньска ѡбласть повѣтъ имаѥтъ оурѧдъ рѣкомъ сєльскъ съвѣтъ ...`
+3. `бѣла роусь сѣи оудѣлъ бѣ члѣнъ ѡбласти рѣкома витєбьска ѡбласть повѣтъ имаѥтъ оурѧдъ рѣкомъ сєльскъ ...`
 
 
 ### Generated Text Samples (Subword-based)
@@ -267,34 +299,34 @@ Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `_инъ_єньсєжапь_н`
-2. `а_вє_шємл҄итѣка_п`
-3. `о_стєрїтовоуспє_`
+1. `_иното_ѥгонокꙑ_ѥ`
+2. `а_одаскꙑ_сточлѣс`
+3. `орлѩꙁа_гокє_ꙁꙑ_с`
 
 **Context Size 2:**
 
-1. `ъ_ка_ѥстъ_бѣлꙗѥтъ`
-2. `и_·_тавѣ_коѩбр҄їꙗ:`
-3. `а_костомолїтарьно`
+1. `ъ_обирѡсьскꙑ_рѣвь`
+2. `и_•_всєли_·_рѡпьс`
+3. `а_посладъпрꙗѥтъ_ꙁ`
 
 **Context Size 3:**
 
-1. `тъ_словѣньскъ_ѥстъ`
-2. `_·_єпїсимь_40_грос`
-3. `ьскъвьсцѣ_на_оупи_`
+1. `тъ_⁙_глаголєптємпє`
+2. `_·_дѣлъ_бѣлороусло`
+3. `ьскъ_прьвовец_гора`
 
 **Context Size 4:**
 
-1. `_ѥстъ_⁙_наи́бѫ́льша_г`
-2. `стъ_гоѵглъ_єси_и_8_`
-3. `ѥстъ_⁙_сѥго_ѩꙁꙑка_к`
+1. `_ѥстъ_·_мїхаилъ_хоу`
+2. `стъ_словєниꙗ_мѫжь_с`
+3. `ѥстъ_⁙_глагоданьска`
 
 
 ### Key Findings
 
 - **Best Predictability:** Context-4 (word) with 97.9% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (77,657 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (77,420 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -310,24 +342,24 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 6,213 |
-| Total Tokens | 63,034 |
-| Mean Frequency | 10.15 |
+| Vocabulary Size | 6,189 |
+| Total Tokens | 62,865 |
+| Mean Frequency | 10.16 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 60.04 |
+| Frequency Std Dev | 60.08 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | и | 2,825 |
-| 2 | ѥстъ | 2,697 |
-| 3 | лѣта | 958 |
-| 4 | бѣ | 912 |
-| 5 | въ | 843 |
-| 6 | градъ | 795 |
-| 7 | ꙁьри | 533 |
-| 8 | такождє | 529 |
+| 1 | и | 2,821 |
+| 2 | ѥстъ | 2,694 |
+| 3 | лѣта | 952 |
+| 4 | бѣ | 910 |
+| 5 | въ | 842 |
+| 6 | градъ | 792 |
+| 7 | ꙁьри | 536 |
+| 8 | такождє | 533 |
 | 9 | жє | 512 |
 | 10 | людии | 470 |
 
@@ -335,39 +367,39 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | статистичьского | 2 |
-| 2 | катєгорїꙗ | 2 |
-| 3 | سخ | 2 |
-| 4 | هس | 2 |
-| 5 | ش | 2 |
-| 6 | ؤخخم | 2 |
-| 7 | خىث | 2 |
-| 8 | ىعةلاثق | 2 |
-| 9 | صشس | 2 |
-| 10 | пльсковьская | 2 |
+| 1 | катєгорїꙗ | 2 |
+| 2 | سخ | 2 |
+| 3 | هس | 2 |
+| 4 | ش | 2 |
+| 5 | ؤخخم | 2 |
+| 6 | خىث | 2 |
+| 7 | ىعةلاثق | 2 |
+| 8 | صشس | 2 |
+| 9 | пльсковьская | 2 |
+| 10 | маѭтъ | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.9368 |
-| R² (Goodness of Fit) | 0.986351 |
+| Zipf Coefficient | 0.9373 |
+| R² (Goodness of Fit) | 0.986343 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 40.9% |
-| Top 1,000 | 72.7% |
+| Top 100 | 41.0% |
+| Top 1,000 | 72.8% |
 | Top 5,000 | 96.2% |
 | Top 10,000 | 0.0% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9864 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 40.9% of corpus
-- **Long Tail:** -3,787 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9863 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 41.0% of corpus
+- **Long Tail:** -3,811 words needed for remaining 100.0% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -383,37 +415,40 @@ Below are text samples generated from each subword-based Markov chain model:
 
 ### 5.1 Cross-Lingual Alignment
 
-> *Note: Multilingual alignment visualization not available for this language.*
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
 
 
 ### 5.2 Model Comparison
 
 | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
 |-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.2996 🏆 | 0.4830 | N/A | N/A |
-| **mono_64d** | 64 | 0.0761 | 0.4499 | N/A | N/A |
-| **mono_128d** | 128 | 0.0111 | 0.4641 | N/A | N/A |
+| **mono_32d** | 32 | 0.2434 | 0.4441 | N/A | N/A |
+| **mono_64d** | 64 | 0.0769 | 0.4495 | N/A | N/A |
+| **mono_128d** | 128 | 0.0128 | 0.4700 | N/A | N/A |
+| **aligned_32d** | 32 | 0.2434 🏆 | 0.4485 | 0.0177 | 0.1032 |
+| **aligned_64d** | 64 | 0.0769 | 0.4699 | 0.0324 | 0.1475 |
+| **aligned_128d** | 128 | 0.0128 | 0.4554 | 0.0442 | 0.1357 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.2996 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.4657. Lower values indicate better semantic separation.
-- **Alignment Quality:** No aligned models evaluated in this run.
+- **Best Isotropy:** aligned_32d with 0.2434 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.4562. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 4.4% R@1 in cross-lingual retrieval.
 - **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
 ## 6.  Morphological Analysis (Experimental)
 
-> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
-
 This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
 
 ### 6.1 Productivity & Complexity
 
 | Metric | Value | Interpretation | Recommendation |
 |--------|-------|----------------|----------------|
-| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
-| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **1.066** | High formulaic/idiomatic content | - |
 
 ### 6.2 Affix Inventory (Productive Units)
 
@@ -422,20 +457,20 @@ These are the most productive prefixes and suffixes identified by sampling the v
 #### Productive Prefixes
 | Prefix | Examples |
 |--------|----------|
-| `-пр` | правилъ, протєстантї́зма, прѣславъ |
-| `-по` | помꙑшлѥниѥ, послѣдни, польꙃєвати |
+| `-по` | поѩла, погꙑнѫли, польꙃєвати |
+| `-пр` | прєждє, придънѣстрии, прасловѣньскъ |
 
 #### Productive Suffixes
 | Suffix | Examples |
 |--------|----------|
-| `-ъ` | въсѣхъ, правилъ, кѷрїллъ |
-| `-къ` | арктїчьскъ, оучєникъ, оукъ |
-| `-ка` | владимѣрьска, банчьска, вльгоградьска |
-| `-нъ` | октадєканъ, ѥдьнѥнъ, дръжавьнъ |
-| `-ска` | владимѣрьска, банчьска, вльгоградьска |
-| `-скъ` | арктїчьскъ, лєниньскъ, въсточьнословѣньскъ |
-| `-кꙑ` | шавьльскꙑ, дрєвл҄ьнѥгрьчьскꙑ, аѵстрїискꙑ |
-| `-ьска` | владимѣрьска, банчьска, вльгоградьска |
+| `-ъ` | въꙁвращєнъ, дѣлъ, ѳєр��пѡнтъ |
+| `-къ` | липьтьскъ, грьчьскъ, словѣньскъ |
+| `-нъ` | въꙁвращєнъ, гла́вьнъ, съꙁиждєнъ |
+| `-ка` | кировьска, фроунꙁєньска, видодъска |
+| `-скъ` | липьтьскъ, грьчьскъ, словѣньскъ |
+| `-ска` | кировьска, фроунꙁєньска, видодъска |
+| `-ьска` | кировьска, фроунꙁєньска, городєньска |
+| `-кꙑ` | блъгарьскꙑ, хръватьскꙑ, словѣньскꙑ |
 
 ### 6.3 Bound Stems (Lexical Roots)
 
@@ -443,18 +478,18 @@ Bound stems are high-frequency subword units that are semantically cohesive but
 
 | Stem | Cohesion | Substitutability | Examples |
 |------|----------|------------------|----------|
-| `боук` | 1.84x | 14 contexts | боукꙑ, боуквꙑ, боукъвъ |
-| `ловѣ` | 1.56x | 18 contexts | словѣ, словѣнъ, словѣнє |
-| `слов` | 1.69x | 14 contexts | слово, слова, словѣ |
-| `ьжав` | 1.70x | 13 contexts | дрьжавѫ, дрьжавꙑ, дрьжавъ |
-| `ньск` | 1.60x | 15 contexts | мѣньска, жєньскъ, мѣньскъ |
-| `ласт` | 1.40x | 20 contexts | власти, властъ, власть |
-| `ьска` | 1.56x | 14 contexts | омьска, людьска, мѣньска |
-| `овѣн` | 1.77x | 9 contexts | словѣнъ, словѣнє, словѣнїꙗ |
-| `град` | 1.57x | 12 contexts | градъ, градѣ, гради |
-| `ьскъ` | 1.55x | 11 contexts | омьскъ, жєньскъ, томьскъ |
-| `блас` | 1.57x | 10 contexts | ѡбласти, область, ѡбласть |
-| `рьжа` | 1.62x | 9 contexts | дрьжавѫ, дрьжавꙑ, дрьжавъ |
+| `боук` | 1.89x | 14 contexts | боукꙑ, боуквꙑ, боукъвь |
+| `ловѣ` | 1.63x | 18 contexts | словѣ, чловѣкъ, словѣнє |
+| `слов` | 1.77x | 14 contexts | слово, словѣ, слова |
+| `ласт` | 1.55x | 20 contexts | властъ, власть, власти |
+| `ьжав` | 1.75x | 13 contexts | дрьжавꙑ, дрьжавъ, дрьжавѫ |
+| `ньск` | 1.65x | 15 contexts | мѣньска, мѣньскъ, жєньскъ |
+| `ьска` | 1.64x | 14 contexts | омьска, єстьска, сѣрьска |
+| `овѣн` | 1.83x | 10 contexts | словѣнє, словѣнъ, словѣнїꙗ |
+| `град` | 1.63x | 13 contexts | градѣ, градъ, гради |
+| `блас` | 1.69x | 10 contexts | ѡбласти, области, ѡбласть |
+| `ьскъ` | 1.63x | 11 contexts | омьскъ, римьскъ, ꙁємьскъ |
+| `рьжа` | 1.69x | 9 contexts | дрьжавꙑ, дрьжавъ, дрьжавѫ |
 
 ### 6.4 Affix Compatibility (Co-occurrence)
 
@@ -462,16 +497,16 @@ This table shows which prefixes and suffixes most frequently co-occur on the sam
 
 | Prefix | Suffix | Frequency | Examples |
 |--------|--------|-----------|----------|
-| `-пр` | `-ъ` | 34 words | приморьскъ, природьнꙑхъ |
-| `-по` | `-ъ` | 34 words | польꙃоуѭтъ, полѫостровъ |
-| `-по` | `-нъ` | 11 words | подобьнъ, поушькинъ |
-| `-по` | `-тъ` | 7 words | польꙃоуѭтъ, польꙃоуѥтъ |
-| `-по` | `-къ` | 7 words | подъбрадъкъ, пол҄ьскъ |
-| `-по` | `-ка` | 7 words | политика, политическа |
-| `-пр` | `-къ` | 6 words | приморьскъ, прѣꙁъсибирьскъ |
-| `-по` | `-скъ` | 6 words | пол҄ьскъ, подольскъ |
-| `-пр` | `-нъ` | 6 words | природьнъ, прѡтонъ |
-| `-по` | `-ска` | 5 words | политическа, подъкарпатьска |
+| `-по` | `-ъ` | 34 words | побѣдъ, помѣновєнъ |
+| `-пр` | `-ъ` | 34 words | прьвꙑимъ, проливъ |
+| `-по` | `-нъ` | 11 words | помѣновєнъ, посъланъ |
+| `-по` | `-ка` | 7 words | подъкарпатьска, по́л̑ьска |
+| `-по` | `-къ` | 7 words | подъбрадъкъ, подълѣсьскъ |
+| `-по` | `-скъ` | 6 words | подълѣсьскъ, пол҄ьскъ |
+| `-пр` | `-нъ` | 6 words | прѣданъ, природьнъ |
+| `-пр` | `-къ` | 6 words | приморьскъ, прьвотравєньскъ |
+| `-по` | `-ска` | 5 words | подъкарпатьска, по́л̑ьска |
+| `-по` | `-ьскъ` | 5 words | подълѣсьскъ, пол҄ьскъ |
 
 ### 6.5 Recursive Morpheme Segmentation
 
@@ -479,26 +514,28 @@ Using **Recursive Hierarchical Substitutability**, we decompose complex words in
 
 | Word | Suggested Split | Confidence | Stem |
 |------|-----------------|------------|------|
-| самостоꙗтѣл҄ьнъ | **`самостоꙗтѣл҄ь-нъ`** | 4.5 | `самостоꙗтѣл҄ь` |
+| гєѡргїиска | **`гєѡргїи-ска`** | 4.5 | `гєѡргїи` |
+| посєлѥниѥ | **`по-��єлѥниѥ`** | 4.5 | `сєлѥниѥ` |
+| октѡврїиска | **`октѡврїи-ска`** | 4.5 | `октѡврїи` |
+| посєлѥниꙗ | **`по-сєлѥниꙗ`** | 4.5 | `сєлѥниꙗ` |
+| самостоꙗтєл҄ьна | **`самостоꙗтєл҄ь-на`** | 4.5 | `самостоꙗтєл҄ь` |
 | аѵстралїиска | **`аѵстралїи-ска`** | 4.5 | `аѵстралїи` |
-| аѵстрїиска | **`аѵстрїи-ска`** | 4.5 | `аѵстрїи` |
-| франкїиска | **`франкїи-ска`** | 4.5 | `франкїи` |
+| самостоꙗтѣльна | **`самостоꙗтѣль-на`** | 4.5 | `самостоꙗтѣль` |
 | аѵстрїискъ | **`аѵстрїи-скъ`** | 4.5 | `аѵстрїи` |
-| сибирьскъ | **`сибирь-скъ`** | 4.5 | `сибирь` |
-| їталїискъ | **`їталїи-скъ`** | 4.5 | `їталїи` |
-| ꙗпѡнїискъ | **`ꙗпѡнїи-скъ`** | 4.5 | `ꙗпѡнїи` |
-| ꙗпѡнїиска | **`ꙗпѡнїи-ска`** | 4.5 | `ꙗпѡнїи` |
-| посєлєниѥ | **`по-сєлєниѥ`** | 4.5 | `сєлєниѥ` |
-| посєлєниꙗ | **`по-сєлєниꙗ`** | 4.5 | `сєлєниꙗ` |
-| поминаѭтъ | **`по-минаѭ-тъ`** | 3.0 | `минаѭ` |
-| подълѣсьскъ | **`по-дълѣсь-скъ`** | 3.0 | `дълѣсь` |
-| єѯадєканъ | **`єѯадє-ка-нъ`** | 3.0 | `єѯадє` |
-| политическꙑ | **`по-литиче-скꙑ`** | 3.0 | `литиче` |
+| приморьскъ | **`пр-имор-ьскъ`** | 3.0 | `имор` |
+| подольскъ | **`по-доль-скъ`** | 3.0 | `доль` |
+| полїтїчьскъ | **`по-лїтїч-ьскъ`** | 3.0 | `лїтїч` |
+| подъꙁємьнъ | **`по-дъꙁємь-нъ`** | 3.0 | `дъꙁємь` |
+| прѣѥмьникъ | **`пр-ѣѥмьни-къ`** | 3.0 | `ѣѥмьни` |
+| потрѣбьна | **`по-трѣбь-на`** | 3.0 | `трѣбь` |
+| политическа | **`по-литиче-ска`** | 3.0 | `литиче` |
 
 ### 6.6 Linguistic Interpretation
 
 > **Automated Insight:**
-The language CU appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+The language Church Slavic shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+> **Note on Idiomaticity:** The high Idiomaticity Gap suggests a large number of frequent multi-word expressions or formulaic sequences that are statistically distinct from their component parts.
 
 ---
 ## 7. Summary & Recommendations
@@ -725,4 +762,4 @@ MIT License - Free for academic and commercial use.
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-03 10:39:18*
+*Report Date: 2026-01-03 20:59:44*

models/embeddings/aligned/cu_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70955bffb6c7be3d460ab37ab99c40c6e50f57e778afffe5c66463f7d02f4c98
+size 1026168646

models/embeddings/aligned/cu_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "cu", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/cu_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2cdfcacbc535058b28d6a343f802b99aa2ba19ed5e169a8e800c130504bad0ec
+size 65664

models/embeddings/aligned/cu_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "cu",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 339,
+  "vocab_size": 2074
+}

models/embeddings/aligned/cu_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ead4a44192bbe83de7d9b363ee83835a8dd73ba2f06a04090d4261e7ffee25bf
+size 256575814

models/embeddings/aligned/cu_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "cu", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/cu_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:885bc7e6b0aba665ab4416a5b9261448be3aed77e6e445dd8d129854b8f6173a
+size 4224

models/embeddings/aligned/cu_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "cu",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 339,
+  "vocab_size": 2074
+}

models/embeddings/aligned/cu_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2dde8a72af6e55db50167090622b22b1b791150967af8dcd91220c45f18147de
+size 513106758

models/embeddings/aligned/cu_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "cu", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/cu_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48f4625de4927eb6c16995c71459ef48aa6fedc42f2ee6b5c236ace594e84724
+size 16512

models/embeddings/aligned/cu_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "cu",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 339,
+  "vocab_size": 2074
+}

models/embeddings/monolingual/cu_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8f180195153a0e4578f44c1f6b1c3eda4c3a7f0672314c1b3ce6aebed01755be
-size 1026177089
+oid sha256:70955bffb6c7be3d460ab37ab99c40c6e50f57e778afffe5c66463f7d02f4c98
+size 1026168646

models/embeddings/monolingual/cu_128d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 128
   },
-  "vocab_size": 2082
+  "vocab_size": 2074
 }

models/embeddings/monolingual/cu_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f90429875eb75105fa8f12361ebaf97112dbdb9c2a3a585b6a0f5093c23fa4a9
-size 256578113
+oid sha256:ead4a44192bbe83de7d9b363ee83835a8dd73ba2f06a04090d4261e7ffee25bf
+size 256575814

models/embeddings/monolingual/cu_32d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 32
   },
-  "vocab_size": 2082
+  "vocab_size": 2074
 }

models/embeddings/monolingual/cu_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:13f1bd4d35a86f2cf0d70eb7c8c834bc4615d38220adae9405e7fddc748a894a
-size 513111105
+oid sha256:2dde8a72af6e55db50167090622b22b1b791150967af8dcd91220c45f18147de
+size 513106758

models/embeddings/monolingual/cu_64d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 64
   },
-  "vocab_size": 2082
+  "vocab_size": 2074
 }

models/subword_markov/cu_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4e949a197cc604e7c761bd6bc85ce653be6b8e3a354f4cd10bcb179122c7c1bc
-size 61462
+oid sha256:ce1edbc4ff3d89a5297689a621049645e7677368c7fb9b48728cbb25ec12c6c9
+size 58311

models/subword_markov/cu_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "cu",
-  "unique_contexts": 1078,
-  "total_transitions": 515604
+  "unique_contexts": 1077,
+  "total_transitions": 514006
 }

models/subword_markov/cu_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b941955b7b2c0d840400a334ec728d235f18de03f0b7d327b53a4c3fc2b0affd
-size 241073
+oid sha256:5afdfa50dfee4f62d0879905cf4838a1b3d40c81ab9b41b63d6ea6cd457392a8
+size 239107

models/subword_markov/cu_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "cu",
-  "unique_contexts": 7639,
-  "total_transitions": 514286
+  "unique_contexts": 7633,
+  "total_transitions": 512684
 }

models/subword_markov/cu_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f131478e09b8f63ad301796670764859e6dfa836f6b9d6ca0311e6ee4f91d06c
-size 678460
+oid sha256:90f1b1cf9de0b5de2daac2e64cce22d8d3bad88762ccaa9bb48cf47e57035ac3
+size 672469

models/subword_markov/cu_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "cu",
-  "unique_contexts": 31947,
-  "total_transitions": 512968
+  "unique_contexts": 31900,
+  "total_transitions": 511362
 }

models/subword_markov/cu_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:56b15f422fd056498e6c88d7059108ebf1372e799ff8caeeefcadf86d5c18d69
-size 1344662
+oid sha256:aa58cd0ccf4b588c6e592483927a99e9989ca8ccc80a23375416ca43d4c3e14b
+size 1340422

models/subword_markov/cu_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "cu",
-  "unique_contexts": 77657,
-  "total_transitions": 511650
+  "unique_contexts": 77420,
+  "total_transitions": 510040
 }

models/subword_ngram/cu_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2126e3213739aec87a6da0c10a5370c2aa9b1ac282cc16a13ff6bb7203679f6e
-size 33800
+oid sha256:933a47acb9a1fe04cf5f8c339161bfde593aa2fd5b818de54fb7999685005a82
+size 33490

models/subword_ngram/cu_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "cu",
-  "unique_ngrams": 2626,
-  "total_ngrams": 515604
+  "unique_ngrams": 2622,
+  "total_ngrams": 514006
 }

models/subword_ngram/cu_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b2cc8ff85bcc54e2024c90771fec3887648656c77e0464212ec94ebfeea2edc3
-size 160670
+oid sha256:13bdd3020e3be644849cb639166b11a14f0d83c1282fab06e46198408ad6d103
+size 160334

models/subword_ngram/cu_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "cu",
-  "unique_ngrams": 12321,
-  "total_ngrams": 514286
+  "unique_ngrams": 12286,
+  "total_ngrams": 512684
 }

models/subword_ngram/cu_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6e5f7ca57181d7dc4617a367f4ab643220e33820321439ffab3d419cf589a2a4
-size 464092
+oid sha256:46deafa006f2e1ae5c400359ae38e82643726ec70519c36b25a393d843cc864f
+size 454137

models/subword_ngram/cu_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "cu",
-  "unique_ngrams": 33307,
-  "total_ngrams": 512968
+  "unique_ngrams": 33187,
+  "total_ngrams": 511362
 }

models/subword_ngram/cu_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e47b0a6dacdf5ec7fb74ed8a879e7c717639ec579260d92137c1833b2adc5b99
+size 647025

models/subword_ngram/cu_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "cu",
+  "unique_ngrams": 46031,
+  "total_ngrams": 510040
+}

models/tokenizer/cu_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:97e6359a0f9b3b36740eb66281c9d74b4750ea93b6ae84d5b6666d1d0cb350c4
-size 574583
+oid sha256:54fcb6b0322245c2814ee47ec98d61ba57fb6cf34a9a04b1300e24ef4e09195a
+size 574629

models/tokenizer/cu_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:64b3f6f09166812fbbb56c670818df981292963a9f019e1f0adfac21a18679da
-size 960296
+oid sha256:2bb8f76ddc298d8534b123428e82c239b77c35db3f1c65d128555ffda988a2bf
+size 960739

models/tokenizer/cu_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:acc38a0fece8f1fa677a20fabe3d2d86630c9a3df52c07ec34dbad75382cb37a
-size 398366
+oid sha256:bbc737e0003034cedb55b6f7127462b442f563eace2f98a46b5dfee83b21875f
+size 398308

models/vocabulary/cu_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:430483667c6dfc4d8c34b266ab0e2f51a877989023eebb849030861107cfd909
-size 111841
+oid sha256:e6f3631f2d06984dc24e1e85fd9b488bc9622b27e62320507f6303dd21c6b88c
+size 110897

models/vocabulary/cu_vocabulary_metadata.json

@@ -1,17 +1,17 @@
 {
   "language": "cu",
-  "vocabulary_size": 6213,
+  "vocabulary_size": 6189,
   "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.24926341379629263,
+    "type_token_ratio": 0.24925489780509452,
     "coverage": {
-      "top_100": 0.34099647231360736,
-      "top_1000": 0.6052426440471944,
-      "top_5000": 0.8007715988214621,
-      "top_10000": 0.8828596720705009
+      "top_100": 0.3413164134423059,
+      "top_1000": 0.6060164518564635,
+      "top_5000": 0.80122660379108,
+      "top_10000": 0.8832077146225478
     },
-    "hapax_count": 12653,
-    "hapax_ratio": 0.6706774090957278,
-    "total_documents": 1318
+    "hapax_count": 12628,
+    "hapax_ratio": 0.6710952861773928,
+    "total_documents": 1322
   }
 }

models/word_markov/cu_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:aa5618d23ca23ed6f616602577c65e3e186ba372648b13a91d6915bcb3a1bca1
-size 679909
+oid sha256:e158acd99119e6c0a3a722c0a7569dd852b3557e8bbb1bccab6a895c47250485
+size 677558

models/word_markov/cu_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "cu",
-  "unique_contexts": 18795,
-  "total_transitions": 74369
+  "unique_contexts": 18746,
+  "total_transitions": 74171
 }

models/word_markov/cu_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e53c547cb424d7129f42e0810a4886b3e115522fb5db795757ffaeff957e9df6
-size 1303006
+oid sha256:f9ab422ae98a7adcbf1a54254ea783e82761b073b30443fa4e0ca363abbf17ea
+size 1299365

models/word_markov/cu_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "cu",
-  "unique_contexts": 48721,
-  "total_transitions": 73051
+  "unique_contexts": 48473,
+  "total_transitions": 72849
 }

models/word_markov/cu_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e1c4cf8e3dc9462e981f1ecec515a2d1249d9723320718db76503ee3b51ce571
-size 1658703
+oid sha256:b14683206db82be572903e01bee66092772026bb58cc5a9818fb9ae0ba5b1059
+size 1649938

models/word_markov/cu_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "cu",
-  "unique_contexts": 58656,
-  "total_transitions": 71733
+  "unique_contexts": 58365,
+  "total_transitions": 71527
 }