fix: make `eos_token`/`pad_token` overridable and add `pickle` support c6d8d63 verified jon-tow commited on Jan 23, 2024
feat(tokenizer): expose merge ranks and special tokens for GGUF 478d491 verified jon-tow commited on Jan 19, 2024