bench-labs developed **GCTokenizer-v1**, which is a multi-lingual tokenizer Available in four sizes: 32K, 65K, 131K and 262K tokens "S, M, L, XL" It utilizes an encoding scheme which allows it to handle characters in any language around the world
General (multi lingual) Consensus (from multiple model tokenizers consensus) Tokenizer
We included an implementation script too, built like BPE- it can encode arbitrary text, most of the time, efficiently
Our analysis explores some implications of this design choice, especially from a distillation perspective where keeping the expert compact could be key for efficient deployment. 🧠