File size: 361 Bytes
23c6c52 3e2fb52 23c6c52 |
1 2 3 4 5 6 7 8 9 10 11 |
---
license: cc0-1.0
datasets:
- code_search_net
library_name: transformers
tags:
- text-generation
- code
- python
---
This is an adapted tokenizer from GPT2 that can recognize tokens to do with Python coding. It is part of the [huggingfaceNLP course exercise](https://huggingface.co/learn/nlp-course/chapter6/2). It uses the method `train_new_from_iterator()` |