add model

README.md

@@ -0,0 +1,68 @@
+---
+
+language: ja
+
+license: cc-by-sa-4.0
+
+datasets:
+
+- wikipedia
+
+widget:
+
+- text: 東京大学で[MASK]の研究をしています。
+
+---
+
+# ELECTRA base Japanese generator
+
+This is a [ELECTRA](https://github.com/google-research/electra) model pretrained on texts in the Japanese language.
+
+The codes for the pretraining are available at [retarfi/language-pretraining](https://github.com/retarfi/language-pretraining/tree/v1.0).
+
+## Model architecture
+
+The model architecture is the same as ELECTRA base in the [original ELECTRA implementation](https://github.com/google-research/electra); 12 layers, 256 dimensions of hidden states, and 4 attention heads.
+
+## Training Data
+
+The models are trained on the Japanese version of Wikipedia.
+
+The training corpus is generated from the Japanese version of Wikipedia, using Wikipedia dump file as of June 1, 2021. 
+
+The corpus file is 2.9GB, consisting of approximately 20M sentences.
+
+## Tokenization
+
+The texts are first tokenized by MeCab with IPA dictionary and then split into subwords by the WordPiece algorithm.
+
+The vocabulary size is 32768.
+
+## Training
+
+The models are trained with the same configuration as ELECTRA base in the [original ELECTRA paper](https://arxiv.org/abs/2003.10555) except size; 512 tokens per instance, 256 instances per batch, and 766k training steps.
+
+The size of the generator is the same of the discriminator.
+
+## Citation
+
+**There will be another paper for this pretrained model. Be sure to check here again when you cite.**
+
+```
+@inproceedings{bert_electra_japanese,
+  title = {Construction and Validation of a Pre-Trained Language Model
+Using Financial Documents}
+  author = {Masahiro Suzuki and Hiroki Sakaji and Masanori Hirano and Kiyoshi Izumi},
+  month = {oct},
+  year = {2021},
+  booktitle = {"Proceedings of JSAI Special Interest Group on Financial Infomatics (SIG-FIN) 27"}
+}
+```
+
+## Licenses
+
+The pretrained models are distributed under the terms of the [Creative Commons Attribution-ShareAlike 4.0](https://creativecommons.org/licenses/by-sa/4.0/).
+
+## Acknowledgments
+
+This work was supported by JSPS KAKENHI Grant Number JP21K12010.

config.json

@@ -0,0 +1,27 @@
+{
+  "architectures": [
+    "ElectraForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "embedding_size": 768,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 256,
+  "initializer_range": 0.02,
+  "intermediate_size": 1024,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "electra",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "tokenizer_class": "BertJapaneseTokenizer",
+  "position_embedding_type": "absolute",
+  "summary_activation": "gelu",
+  "summary_last_dropout": 0.1,
+  "summary_type": "first",
+  "summary_use_proj": true,
+  "transformers_version": "4.7.0",
+  "type_vocab_size": 2,
+  "vocab_size": 32768
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c58bfa60829cdaa825662f71d565b1ff9fc964020fab94108cadebf82cf099b
+size 141960100

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tokenizer_config.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "do_lower_case": false, "do_word_tokenize": true, "do_subword_tokenize": true, "word_tokenizer_type": "mecab", "subword_tokenizer_type": "wordpiece", "never_split": null, "mecab_kwargs": {"mecab_dic": "ipadic"}, "tokenize_chinese_chars": false}