| T5 is a more unique model that casts all NLP tasks into a text-to-text problem using specific prefixes. For example, the prefix Summarize: indicates a summarization task. T5 is pretrained by supervised (GLUE and SuperGLUE) training and self-supervised training (randomly sample and drop out 15% of tokens). | |
| Audio | |
| Encoder[[audio-encoder]] | |
| Wav2Vec2 uses a Transformer encoder to learn speech representations directly from raw audio waveforms. |