docs: add info to model card

README.md

@@ -1 +1,101 @@
-## deep-metal
+## Model description
+
+**DeepMetal** is a model capable of generating lyrics taylored for heavy metal songs.
+The model is based on the [OpenAI GPT-2](https://huggingface.co/gpt2) and has been finetuned on a dataset of 141,718 heavy metal songs lyrics.
+
+### Legal notes
+
+Due to incertainity about legal rights, the dataset used for training the model is not provided. I hope you'll understand. The lyrics in question have been scraped from scraped from [DarkLyrics](http://www.darklyrics.com/) using the library [metal-parser](https://github.com/lucone83/metal-parser).
+
+## Intended uses and limitations
+
+The model is released under the [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0). You can use the raw model for lyrics generation or fine-tune it further to a downstream task.
+
+## How to use
+
+You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, it could be good to set a seed for reproducibility:
+
+```python
+>>> from transformers import pipeline, set_seed
+>>> generator = pipeline('text-generation', model='lucone83/deep-metal', device=-1)  # to use GPU, set device=<CUDA_device_ordinal>
+>>> set_seed(42)
+>>> generator(
+        "I'll kill you and your dreams tonight",
+        num_return_sequences=1,
+        max_length=256,
+        min_length=128,
+        top_p=0.97,
+        top_k=0,
+        temperature=0.90
+    )
+[{'generated_text': "I'll kill you and your dreams tonight\nYou're already dead\nI'll kill you and your dreams tonight\nNow I'm done\nNo words to say for you\nAnd don't let me down\nYou'll leave with nothing\nNo one could to wait for you\nYou were already dead\nYou were already dead\n\nI want to know what did you do?\nWhen you tried to run away\nI'll kill you and your dreams tonight\nYou're already dead\nI'll kill you and your dreams tonight\nNow I'm done\nNo words to say for you\nAnd don't let me down\nYou'll leave with nothing\nNo one could to wait for you\nYou were already dead\nYou were already dead "}]
+```
+
+Of course, it's possible to play with parameters like `top_k`, `top_p`, `temperature`, `max_length` and all the other parameters included in the `generate` method. Please look at the [documentation](https://huggingface.co/transformers/main_classes/model.html?highlight=generate#transformers.generation_utils.GenerationMixin.generate) for further insights.
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import GPT2Tokenizer, GPT2Model
+
+tokenizer = GPT2Tokenizer.from_pretrained('lucone83/deep-metal')
+model = GPT2Model.from_pretrained('lucone83/deep-metal')
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+output_features = model(**encoded_input)
+```
+
+and in TensorFlow:
+
+```python
+from transformers import GPT2Tokenizer, TFGPT2Model
+
+tokenizer = GPT2Tokenizer.from_pretrained('lucone83/deep-metal')
+model = TFGPT2Model.from_pretrained('lucone83/deep-metal')
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='tf')
+output_features = model(encoded_input)
+```
+
+## Model training
+
+The dataset used for training this model contained 141,718 heavy metal songs lyrics.
+The model has been trained using an NVIDIA Tesla T4 with 16 GB, using the following command:
+
+```bash
+python run_language_modeling.py \
+    --output_dir=$OUTPUT_DIR \
+    --model_type=gpt2 \
+    --model_name_or_path=gpt2 \
+    --do_train \
+    --train_data_file=$TRAIN_FILE \
+    --do_eval \
+    --eval_data_file=$VALIDATION_FILE \
+    --per_device_train_batch_size=3 \
+    --per_device_eval_batch_size=3 \
+    --evaluate_during_training \
+    --learning_rate=1e-5 \
+    --num_train_epochs=20 \
+    --logging_steps=3000 \
+    --save_steps=3000 \
+    --gradient_accumulation_steps=3
+```
+
+To checkout the code related to training and testing, please look at the [GitHub repository](https://github.com/lucone83/deep-metal) of the project.
+
+## Evaluation results
+
+The model achieves the following results:
+
+```bash
+{
+    'eval_loss': 3.0047452173826406,
+    'epoch': 19.99987972095261,
+    'total_flos': 381377736125448192,
+    'step': 55420
+}
+
+perplexity = 20.18107365414611
+```
+
+![eval-loss](https://github.com/lucone83/deep-metal/blob/master/resources/deep-metal-eval-loss.png?raw=true)

config.json

@@ -4,7 +4,7 @@
     "GPT2LMHeadModel"
   ],
   "attn_pdrop": 0.1,
-  "bos_token_id": 50256,
+  "bos_token_id": 50257,
   "embd_pdrop": 0.1,
   "eos_token_id": 50256,
   "initializer_range": 0.02,
@@ -25,7 +25,8 @@
   "task_specific_params": {
     "text-generation": {
       "do_sample": true,
-      "max_length": 1024
+      "max_length": 256,
+      "min_length": 128
     }
   },
   "total_flos": 371606622276943872,