---
license: apache-2.0
datasets:
- pkupie/mc2_corpus
language:
- bo
- ug
- kk
- mn
- zh
base_model:
- hfl/cino-base-v2
---
# XLM-SWCM: Multilingual Encoder with Shared Weights Pretraining

## Overview

XLM-SWCM (Cross-lingual Language Model with Shared Weights Cross-lingual Modeling) is an innovative sequence-to-sequence model specifically designed to address the challenges of extremely low-resource languages. Our framework introduces a novel weight-sharing mechanism between encoder and decoder components, enabling effective knowledge transfer from multilingual encoders to generation tasks.

## Key Innovations

* **Shared Weight Framework**: Strategic weight reuse between encoder and decoder layers
* **Hybrid Decoder Architecture**: Combines:
  * Standard transformer decoder layers
  * Custom decoder layers with dual FFN structure
  * Optimized layer insertion pattern (1 normal layer per 3 custom layers)
* **Efficient Adaptation**: Enables effective text generation with minimal training data

## Model Architecture


| Component      | Description                                                         |
| -------------- | ------------------------------------------------------------------- |
| **Encoder**    | XLM-RoBERTa base (CINO v2 variant)                                  |
| **Decoder**    | Hybrid transformer with:                                            |
|                | • NormalDecoderLayer: Randomly initialized standard layers         |
|                | • CustomDecoderLayer: Weight-shared layers with dual FFN structure |
| **Parameters** | 492M total parameters                                               |

### Advanced Features

* Beam search decoding
* Mixed-precision training
* Cross-lingual transfer learning

For detailed usage instructions, see our [GitHub repository](https://github.com/asd765973346/xlm-swcm)

## Supported Languages

Primary focus on Chinese minority languages:

* Tibetan (bo)
* Uyghur (ug)
* Kazakh (kk)
* Mongolian (mn)
* Chinese (zh)

## Citation

```
@article{su2025multilingualencoderknowsrealize,
  author       = {Zeli Su and Ziyin Zhang and Guixian Xu and Jianing Liu and Xu Han and Ting Zhang and Yushuang Dong},
  title        = {Multilingual Encoder Knows more than You Realize: Shared Weights Pretraining
                  for Extremely Low-Resource Languages},
  journal      = {CoRR},
  volume       = {abs/2502.10852},
  year         = {2025},
  url          = {https://doi.org/10.48550/arXiv.2502.10852},
  doi          = {10.48550/ARXIV.2502.10852},
  eprinttype    = {arXiv},
  eprint       = {2502.10852}
}