| This can be done by directly feeding the sentence to the tokenizer, which leverages the Rust implementation of 🤗 Tokenizers for peak performance. | |
| thon | |
| inputs = tokenizer(sequence) | |
| The tokenizer returns a dictionary with all the arguments necessary for its corresponding model to work properly. |