lopentu/Taiwan-Lawstral

Model Details

• Continually pretrained from MediaTek-Research/Breeze-7B-32k-Base-v1_0
• Base model with no instruction fine-tuning

Training Details

• Hardware: 2 NVIDIA DGX H100 nodes on Taipei-1
• Software: NVIDIA NeMo and NVIDIA NeMo Framework Launcher
• Training time: 7 days
• Context length: 8192 tokens
• Steps: 81783
• Global batch size: 112
• Epochs: 3
• Tokens trained on: 75B+
• Learning rate:
  • Schedule: Cosine
  • Warmup steps: 107
  • Constant steps: 11873
  • Max LR: 1e-4
  • Min LR: 1e-5

Dataset Details

• Trained mostly on Taiwan legal documents
• Tokens: 25B