| license: cc0-1.0 | |
| datasets: | |
| - code_search_net | |
| library_name: transformers | |
| tags: | |
| - text-generation | |
| - code | |
| - python | |
| This is an adapted tokenizer from GPT2 that can recognize tokens to do with Python coding. It is part of the [huggingfaceNLP course exercise](https://huggingface.co/learn/nlp-course/chapter6/2). It uses the method `train_new_from_iterator()` |