thisisHJLee's picture
add tokenizer
5726c4f
raw
history blame contribute delete
772 Bytes
{"\n": 0, "0": 2, "1": 3, "2": 4, "3": 5, "4": 6, "5": 7, "6": 8, "7": 9, "8": 10, "9": 11, "a": 12, "b": 13, "c": 14, "h": 15, "l": 16, "m": 17, "n": 18, "o": 19, "p": 20, "s": 21, "t": 22, "z": 23, "γ„±": 24, "γ„²": 25, "γ„΄": 26, "γ„΅": 27, "γ„Ά": 28, "γ„·": 29, "γ„Έ": 30, "γ„Ή": 31, "γ„Ί": 32, "γ„»": 33, "γ„Ό": 34, "γ„Ύ": 35, "γ…€": 36, "ㅁ": 37, "γ…‚": 38, "γ…ƒ": 39, "γ…„": 40, "γ……": 41, "γ…†": 42, "γ…‡": 43, "γ…ˆ": 44, "γ…‰": 45, "γ…Š": 46, "γ…‹": 47, "γ…Œ": 48, "ㅍ": 49, "γ…Ž": 50, "ㅏ": 51, "ㅐ": 52, "γ…‘": 53, "γ…’": 54, "γ…“": 55, "γ…”": 56, "γ…•": 57, "γ…–": 58, "γ…—": 59, "γ…˜": 60, "γ…™": 61, "γ…š": 62, "γ…›": 63, "γ…œ": 64, "ㅝ": 65, "γ…ž": 66, "γ…Ÿ": 67, "γ… ": 68, "γ…‘": 69, "γ…’": 70, "γ…£": 71, ",": 72, "|": 1, "[UNK]": 73, "[PAD]": 74}