Request for Guidance on Generating tokenizer.model from tokenizer.json
#216
by
kimppa
- opened
Hi,
I'm working with the Meta-Llama-3-8B model and I'm trying to figure out how the "tokenizer.model" file was generated from "tokenizer.json" and "tokenizer_config.json".
If you could share the code or script that was used, that would be great.
If not, I'd appreciate some details on the process.
Thanks!