metadata
language:
- en
library_name: pytorch
tags:
- language-model
- gpt2
- transformer
- wikitext-103
model-index:
- name: gpt2_wt103-40m_12-layer
results:
- task:
type: language-modeling
dataset:
type: wikitext
name: Wikitext-103
metrics:
- type: perplexity
value: 40.3
Model description
paper: Characterizing Verbatim Short-Term Memory in Neural Language Models
This is a gpt2-small-like decoder-only transformer model trained on a 40M token subset of the wikitext-103 dataset.
Intended uses
This checkpoint is intended for research purposes, for example those interested in studying the behavior of transformer language models trained on smaller datasets.