liwii
/

line-distilbert-base-japanese-fork

Model card Files Files and versions Community

line-distilbert-base-japanese-fork / distilbert_japanese_tokenizer.py

Commit History

Fix following https://github.com/huggingface/transformers/commit/2da8853775b61cde0894dee17c6c713aba711688

08b2a7b

kokiryu commited on Dec 1, 2023

Allow only unidic_lite (#2)

b6c9038

kajyuuen commited on Mar 22, 2023

Improve tokenizers (#1)

0d49f09

kajyuuen commited on Mar 22, 2023

Initial commit

0cfbc01

kajyuuen commited on Mar 10, 2023