a31413510 commited on
Commit
59f7e5a
·
verified ·
1 Parent(s): 61cd5cf

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +8 -6
README.md CHANGED
@@ -24,7 +24,7 @@ tags:
24
  </div>
25
 
26
  <div align="center" style="line-height: 1;">
27
- <a href="LICENSE" style="margin: 2px;">
28
  <img alt="License" src="https://img.shields.io/badge/License-Apache2.0-A5de54" style="display: inline-block; vertical-align: middle;"/>
29
  </a>
30
  </div>
@@ -35,11 +35,11 @@ tags:
35
 
36
  The advanced capabilities of the ERNIE 4.5 models, particularly the MoE-based A47B and A3B series, are underpinned by several key technical innovations:
37
 
38
- 1. **Multimodal Heterogeneous MoE Pre-Training**: Our models are jointly trained on both textual and visual modalities to better capture the nuances of multimodal information and improve performance on tasks involving text generation, image understanding, and cross-modal reasoning. To achieve this without one modality hindering the learning of another, we designed a *heterogeneous MoE structure*, incorporated *modality-isolated routing*, and employed *router orthogonal loss* and *multimodal token-balanced loss*. These architectural choices ensure that both modalities are effectively represented, allowing for mutual reinforcement during training.
39
 
40
- 2. **Scaling-Efficient Infrastructure**: We propose a novel heterogeneous hybrid parallelism and hierarchical load balancing strategy for efficient training of ERNIE 4.5 models. By using intra-node expert parallelism, memory-efficient pipeline scheduling, FP8 mixed-precision training and fine-grained recomputation methods, we achieve remarkable pre-training throughput. For inference, we propose Multi-Expert Parallel Collaboration method and Convolutional Code Quantization algorithm to achieve 4-bit/2-bit lossless quantization. Furthermore, we introduce PD disaggregation with dynamic role switching for effective resource utilization to enhance inference performance for ERNIE 4.5 MoE models. Built on [PaddlePaddle](https://github.com/PaddlePaddle/Paddle), ERNIE 4.5 delivers high-performance inference across a wide range of hardware platforms.
41
 
42
- 3. **Modality-Specific Post-training**: To meet the diverse requirements of real-world applications, we fine-tuned variants of the pretrained model for specific modalities. Our *LLMs* are optimized for general-purpose language understanding and generation. The *VLMs* focuses on visual-language understanding and supports both thinking and no-thinking mode. Each model employed a combination of *Supervised Fine-tuning (SFT)* *Direct Preference Optimization (DPO)* or a modified reinforcement learning method named *Unified Preference Optimization (UPO)* for post-training.
43
 
44
  ## Model Overview
45
 
@@ -63,10 +63,12 @@ ERNIE-4.5-0.3B-Base is a text dense Base model. The following are the model conf
63
  Usage Examples:
64
 
65
  ```bash
 
 
66
  # SFT
67
- erniekit train --stage SFT --model_name_or_path baidu/ERNIE-4.5-0.3B-Base-Paddle --train_dataset_path your_dataset_path
68
  # DPO
69
- erniekit train --stage DPO --model_name_or_path baidu/ERNIE-4.5-0.3B-Base-Paddle --train_dataset_path your_dataset_path
70
  ```
71
 
72
  For more detailed examples, including SFT with LoRA, multi-GPU configurations, and advanced scripts, please refer to the examples folder within the [ERNIEKit](https://github.com/PaddlePaddle/ERNIE) repository.
 
24
  </div>
25
 
26
  <div align="center" style="line-height: 1;">
27
+ <a href="#license" style="margin: 2px;">
28
  <img alt="License" src="https://img.shields.io/badge/License-Apache2.0-A5de54" style="display: inline-block; vertical-align: middle;"/>
29
  </a>
30
  </div>
 
35
 
36
  The advanced capabilities of the ERNIE 4.5 models, particularly the MoE-based A47B and A3B series, are underpinned by several key technical innovations:
37
 
38
+ 1. **Multimodal Heterogeneous MoE Pre-Training:** Our models are jointly trained on both textual and visual modalities to better capture the nuances of multimodal information and improve performance on tasks involving text understanding and generation, image understanding, and cross-modal reasoning. To achieve this without one modality hindering the learning of another, we designed a *heterogeneous MoE structure*, incorporated *modality-isolated routing*, and employed *router orthogonal loss* and *multimodal token-balanced loss*. These architectural choices ensure that both modalities are effectively represented, allowing for mutual reinforcement during training.
39
 
40
+ 2. **Scaling-Efficient Infrastructure:** We propose a novel heterogeneous hybrid parallelism and hierarchical load balancing strategy for efficient training of ERNIE 4.5 models. By using intra-node expert parallelism, memory-efficient pipeline scheduling, FP8 mixed-precision training and finegrained recomputation methods, we achieve remarkable pre-training throughput. For inference, we propose *multi-expert parallel collaboration* method and *convolutional code quantization* algorithm to achieve 4-bit/2-bit lossless quantization. Furthermore, we introduce PD disaggregation with dynamic role switching for effective resource utilization to enhance inference performance for ERNIE 4.5 MoE models. Built on [PaddlePaddle](https://github.com/PaddlePaddle/Paddle), ERNIE 4.5 delivers high-performance inference across a wide range of hardware platforms.
41
 
42
+ 3. **Modality-Specific Post-Training:** To meet the diverse requirements of real-world applications, we fine-tuned variants of the pre-trained model for specific modalities. Our LLMs are optimized for general-purpose language understanding and generation. The VLMs focuses on visuallanguage understanding and supports both thinking and non-thinking modes. Each model employed a combination of *Supervised Fine-tuning (SFT)*, *Direct Preference Optimization (DPO)* or a modified reinforcement learning method named *Unified Preference Optimization (UPO)* for post-training.
43
 
44
  ## Model Overview
45
 
 
63
  Usage Examples:
64
 
65
  ```bash
66
+ # Download Model
67
+ huggingface-cli download baidu/ERNIE-4.5-0.3B-Base-Paddle --local-dir baidu/ERNIE-4.5-0.3B-Base-Paddle
68
  # SFT
69
+ erniekit train examples/configs/ERNIE-4.5-0.3B/sft/run_sft_8k.yaml model_name_or_path=baidu/ERNIE-4.5-0.3B-Base-Paddle
70
  # DPO
71
+ erniekit train examples/configs/ERNIE-4.5-0.3B/dpo/run_dpo_8k.yaml model_name_or_path=baidu/ERNIE-4.5-0.3B-Base-Paddle
72
  ```
73
 
74
  For more detailed examples, including SFT with LoRA, multi-GPU configurations, and advanced scripts, please refer to the examples folder within the [ERNIEKit](https://github.com/PaddlePaddle/ERNIE) repository.