Longxu Dou's picture

Longxu Dou PRO

dreamerdeo

·

https://longxudou.github.io/

AI & ML interests

Natural Language Processing

Recent Activity

reacted to their post with ➕ 1 day ago

🚀 Excited to share our technical report on the Southeast Asian multilingual model Sailor2 and its latest updates! Our 49-page report details Sailor2's development journey, including multilingual data cleaning, small model data mixture simulations, multi-stage continual pre-training, multi-stage post-training, and multi-cultural multi-lingual evaluations. Sailor2 aims to streamline the multilingual model pre-training process efficiently for the community. 🧭 We highlight Sailor2's impressive performance in low-resource language translation scenarios and its cultural understanding advantages in Southeast Asia, promoting practical applications for regional languages. Model updates include: 💡 More precise outputs: Reduced redundancy in model outputs through refined post-training data and optimization techniques. 🌈 Handling longer texts: Expanded to handle up to 128K context length in Southeast Asian languages through long-text training. ⚡️ Faster inference: Achieved 2.5x faster inference speed with speculative decoding. 🌪️ More model sizes: Introduced new sizes of 3B and 14B through model pruning. 🌟 All models are Apache-licensed for commercial use; development tools (code, resources) are open-source. 📚 Technical report: https://huggingface.co/papers/2502.12982 🤖️ Models: https://huggingface.co/collections/sail/sailor2-language-models-674d7c9e6b4dbbd9a869906b 💬 Demo: https://huggingface.co/spaces/sail/Sailor2-20B-Chat 📣 Sailor2 community: https://huggingface.co/sailor2

updated a Space 1 day ago

sailor2/README

new activity 1 day ago

sail/Sailor2-8B-Chat:Fix formatting

View all activity

Organizations

dreamerdeo's activity

upvoted a paper 3 days ago

Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs

Paper • 2502.12982 • Published 3 days ago • 9

upvoted a paper about 1 month ago

Exploring Multi-Grained Concept Annotations for Multimodal Large Language Models

Paper • 2412.05939 • Published Dec 8, 2024 • 16

upvoted 7 collections 3 months ago

Sailor2 Benchmarks

1 item • Updated Dec 3, 2024 • 2

Sailor2 Post-training Datasets

4 items • Updated 5 days ago • 5

Sailor2 Pre-training Datasets

8 items • Updated Dec 4, 2024 • 4

Sailor2 Models

9 items • Updated 5 days ago • 4

🔱 Sailor2 Language Models

Sailing in South-East Asia with Inclusive Multilingual LLMs • 34 items • Updated 3 days ago • 24

OpenCoder Model

OpenCoder Models • 9 items • Updated Nov 19, 2024 • 10

OS-Atlas

OS-Atlas series models • 7 items • Updated Nov 18, 2024 • 13

upvoted an article 7 months ago

Article

RegMix: Data Mixture as Regression for Language Model Pre-training

By

•

Jul 11, 2024

• 11

upvoted a collection 7 months ago

📈 Scaling Laws with Vocabulary

Increase your vocabulary size when you scale up your language model • 5 items • Updated Aug 11, 2024 • 6

upvoted a paper 7 months ago

Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

Paper • 2407.13623 • Published Jul 18, 2024 • 54

upvoted a paper 8 months ago

Bootstrapping Language Models with DPO Implicit Rewards

Paper • 2406.09760 • Published Jun 14, 2024 • 39

upvoted 2 collections 8 months ago

💡 DICE

Self-alignment with DPO Implicit Rewards • 5 items • Updated Jul 28, 2024 • 9

🧬 RegMix: Data Mixture as Regression

Automatic data mixture method for large language model pre-training • 10 items • Updated Jul 26, 2024 • 8

upvoted a paper 8 months ago

RegMix: Data Mixture as Regression for Language Model Pre-training

Paper • 2407.01492 • Published Jul 1, 2024 • 36

upvoted a collection 9 months ago

Datasets for Pretrained Thai LLM

List Datasets for pretrained Thai LLM by PyThaiNLP • 23 items • Updated Sep 12, 2024 • 10

upvoted an article 10 months ago

Article

Large-scale Near-deduplication Behind BigCode

May 16, 2023

• 21

upvoted a collection 10 months ago

Meta Llama 3

This collection hosts the transformers and original repos of the Meta Llama 3 and Llama Guard 2 releases • 5 items • Updated Dec 6, 2024 • 714

upvoted a paper 11 months ago

Sailor: Open Language Models for South-East Asia

Paper • 2404.03608 • Published Apr 4, 2024 • 20