File size: 2,997 Bytes
e79166d 4df46b0 e79166d 0b0286f e79166d 0b0286f e79166d 0b0286f 4df46b0 ed2f740 4df46b0 e79166d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 |
# Ring-lite
<p align="center">
<img src="https://huggingface.co/inclusionAI/Ring-lite-distill-preview/resolve/main/ant-bailing.png" width="100"/>
<p>
<p align="center">
🤗 <a href="https://huggingface.co/inclusionAI">Hugging Face</a>
<p>
## Introduction
Ring-lite is an fully open-source MoE LLM provided by InclusionAI, which has 16.8B parameters with 2.75B activated parameters. It was derived from [Ling-lite-1.5](https://huggingface.co/inclusionAI/Ling-lite-1.5) through a training process involving reasoning SFT, reasoning RL and general SFT. This model delivers performance comparable to [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) on reasoning benchmarks, while activating only one-third of their parameters.
## Model Downloads
<div align="center">
| **Model** | **#Total Params** | **#Activated Params** | **Context Length** | **Download** |
| :----------------: | :---------------: | :-------------------: | :----------------: | :----------: |
| Ring-lite-distill-preview | 16.8B | 2.75B | 64K | [🤗 HuggingFace](https://huggingface.co/inclusionAI/Ring-lite) |
</div>
## Evaluation
In order to fully evaluate the model's reasoning performance, we examined Ring-lite on several reasoning benchmarks, including MATH-500, AIME-24, AIME-24, Livecodebench, Codeforces and GPQA.
<p align="center">
<img src="https://huggingface.co/inclusionAI/Ring-lite/blob/main/performance.pdf" width="100"/>
<p>
More details will be reported in our technical report. [TBD]
## Quickstart
### 🤗 Hugging Face Transformers
Here is a code snippet to show you how to use the chat model with `transformers`:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "inclusionAI/Ring-lite-distill-preview"
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
prompt = "Give me a short introduction to large language models."
messages = [
{"role": "system", "content": "You are Ring, an assistant created by inclusionAI"},
{"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
**model_inputs,
max_new_tokens=8192
)
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
```
## Dataset
The training data of Ring-lite-distill-preview will be released soon.
## Deployment
Please refer to [GitHub](https://github.com/inclusionAI/Ring/blob/main/README.md)
## License
This code repository is licensed under [the MIT License](https://huggingface.co/inclusionAI/Ring-lite-distill/blob/main/LICENSE).
## Citation
[TBD]
|