Update README.md

README.md

@@ -6,9 +6,10 @@ tags:
 ---
 
 ## Why should you use this and not the tiktoken included in the orignal model?
-1. Original tokenizer pad the vocabulary to correct size with `<extra_N>` tokens but encoder never uses them
-2. Original tokenizer use eos as pad token which may confuse trainers to mask out the eos token so model never output eos.
-3. [NOT FIXED: INVESTIGATING] config.json embedding size of "vocab_size": 100352 does not match 100277
+1. This tokenizer is validated with the https://huggingface.co/datasets/xn (all languages) to be encode/decode compatible with dbrx-base tiktoken
+2. Original tokenizer pad the vocabulary to correct size with `<extra_N>` tokens but encoder never uses them
+3. Original tokenizer use eos as pad token which may confuse trainers to mask out the eos token so model never output eos.
+4. [NOT FIXED: INVESTIGATING] config.json embedding size of "vocab_size": 100352 does not match 100277
 
 modified from original code @ https://huggingface.co/Xenova/dbrx-instruct-tokenizer