📈 Scaling Laws with Vocabulary - a sail Collection

sail 's Collections

🚀 Active PRM

🌾Oat-Zero: Understanding R1-Zero-Like Training

🔱 Sailor2 Language Models

🧬 RegMix: Data Mixture as Regression

📈 Scaling Laws with Vocabulary

⚓️ Sailor Language Models

📈 Scaling Laws with Vocabulary

updated Aug 11, 2024

Increase your vocabulary size when you scale up your language model

Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

Paper • 2407.13623 • Published Jul 18, 2024 • 57
Running

11

11

Scaling With Vocab Demo

📊

Predict optimal vocabulary size based on model parameters
sail/scaling-vocab-3b-43k-overtrain

Text Generation • Updated Jul 19, 2024 • 11
sail/scaling-vocab-3b-32k-overtrain

Text Generation • Updated Jul 19, 2024 • 5
sail/scaling-with-vocab-trained-tokenizers

Updated Aug 2, 2024 • 2