thisisHJLee's picture
add tokenizer
3af1aba
raw
history blame contribute delete
752 Bytes
{"\n": 0, "0": 2, "1": 3, "2": 4, "3": 5, "4": 6, "5": 7, "6": 8, "7": 9, "8": 10, "9": 11, "a": 12, "b": 13, "c": 14, "h": 15, "l": 16, "m": 17, "n": 18, "p": 19, "s": 20, "t": 21, "z": 22, "γ„±": 23, "γ„²": 24, "γ„΄": 25, "γ„΅": 26, "γ„Ά": 27, "γ„·": 28, "γ„Έ": 29, "γ„Ή": 30, "γ„Ί": 31, "γ„»": 32, "γ„Ό": 33, "γ…€": 34, "ㅁ": 35, "γ…‚": 36, "γ…ƒ": 37, "γ…„": 38, "γ……": 39, "γ…†": 40, "γ…‡": 41, "γ…ˆ": 42, "γ…‰": 43, "γ…Š": 44, "γ…‹": 45, "γ…Œ": 46, "ㅍ": 47, "γ…Ž": 48, "ㅏ": 49, "ㅐ": 50, "γ…‘": 51, "γ…’": 52, "γ…“": 53, "γ…”": 54, "γ…•": 55, "γ…–": 56, "γ…—": 57, "γ…˜": 58, "γ…™": 59, "γ…š": 60, "γ…›": 61, "γ…œ": 62, "ㅝ": 63, "γ…ž": 64, "γ…Ÿ": 65, "γ… ": 66, "γ…‘": 67, "γ…’": 68, "γ…£": 69, ",": 70, "|": 1, "[UNK]": 71, "[PAD]": 72}