chengzl18
/

cctokenizer

Inference Endpoints

Model card Files Files and versions Community

1 contributor

History: 4 commits

chengzl18's picture

Add EOS special token

a460612 over 1 year ago

.gitattributes

1.48 kB

initial commit over 1 year ago
README.md

28 Bytes

initial commit over 1 year ago
cctokenizer.py

24.1 kB

A Chinese Character Tokenizer with a Many-to-one Mapping. over 1 year ago
config.json

840 Bytes

A Chinese Character Tokenizer with a Many-to-one Mapping. over 1 year ago
replace.json

146 kB

A Chinese Character Tokenizer with a Many-to-one Mapping. over 1 year ago
special_tokens_map.json

149 Bytes

Add EOS special token over 1 year ago
tokenizer_config.json

544 Bytes

A Chinese Character Tokenizer with a Many-to-one Mapping. over 1 year ago
vocab.txt

49.8 kB

A Chinese Character Tokenizer with a Many-to-one Mapping. over 1 year ago