chengzl18
/

cctokenizer

Inference Endpoints

Model card Files Files and versions Community

cctokenizer / special_tokens_map.json

Commit History

Add EOS special token

a460612

chengzl18 commited on Mar 25, 2023

A Chinese Character Tokenizer with a Many-to-one Mapping.

9836307

chengzl18 commited on Mar 24, 2023