Update README.md

README.md

@@ -1,199 +1,262 @@
 ---
-library_name: transformers
-tags: []
+language:
+- en
+tags:
+- babylm-baseline
+- strict-small
+- babylm-2025
 ---
 
-# Model Card for Model ID
+# Model Card for the Preference Optimization Interaction Baseline
 
-<!-- Provide a quick summary of what the model is/does. -->
+<!-- Provide a quick summary of what the model is/does. [Optional] -->
+A 124M model with the GPT-2 architecture trained with the next token prediction loss for 10 epochs (~100M words), as a naive autoregressive baseline for the Strict-Small track of the 2025 BabyLM challenge.
 
 
+#  Table of Contents
 
-## Model Details
+- [Model Card for Strict-Small-track GPT-2 Baseline](#model-card-for--model_id-)
+- [Table of Contents](#table-of-contents)
+- [Model Details](#model-details)
+  - [Model Description](#model-description)
+- [Uses](#uses)
+- [Training Details](#training-details)
+  - [Training Data](#training-data)
+  - [Hyperparameters](#hyperparameters)
+  - [Training Procedure](#training-procedure)
+    - [Size and Checkpoints](#size-and-checkpoints)
+- [Evaluation](#evaluation)
+  - [Testing Data & Metrics](#testing-data-factors--metrics)
+    - [Testing Data](#testing-data)
+    - [Metrics](#metrics)
+  - [Results](#results)
+- [Technical Specifications](#technical-specifications-optional)
+  - [Model Architecture and Objective](#model-architecture-and-objective)
+  - [Compute Infrastructure](#compute-infrastructure)
+    - [Hardware](#hardware)
+    - [Software](#software)
+    - [Training Time](#training-time)
+- [Citation](#citation)
+- [Model Card Authors](#model-card-authors-optional)
+- [Bibliography](#bibliography)
 
-### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
+# Model Details
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+## Model Description
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+<!-- Provide a longer summary of what this model is/does. -->
+This one of the two Strict-track baselines 2025 BabyLM challenge.
 
-### Model Sources [optional]
+- **Developed by:** Mustafa Ömer Gül
+- **Model type:** Causal language model
+- **Language(s) (NLP):** eng
+- **Resources for more information:**
+    - [GitHub Repo](https://github.com/momergul/babylm-gpt2-baseline)
 
-<!-- Provide the basic links for the model. -->
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
+# Uses
 
 <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+This is a pre-trained language model.
+It can be used to evaluate tasks in a zero-shot manner and also can be fine-tuned for downstream tasks.
+It can be used for language generation but given its small size and low number of words trained on, do not expect LLM-level performance.
 
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+# Training Details
 
-[More Information Needed]
+## Training Data
 
-### Recommendations
+<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+We used the BabyLM 10M (Strict-Small) dataset for training. It is composed of the following:
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+| Source | Weight | Domain | Citation | Website | License |
+| --- | --- | --- | --- | --- | --- |
+| BNC | 8% | Dialogue | BNC Consortium (2007) | [link](http://www.natcorp.ox.ac.uk/) | [link](http://www.natcorp.ox.ac.uk/docs/licence.html) <sup>1</sup> |
+| CHILDES | 29% | Dialogue, Child-Directed | MacWhinney (2000) | | [link](https://talkbank.org/share/rules.html) |
+| Project Gutenberg | 26% | Fiction, Nonfiction | Gerlach & Font-Clos (2020) | [link](https://github.com/pgcorpus/gutenberg) | [link](https://www.gutenberg.org/policy/license.html) |
+| OpenSubtitles | 20% | Dialogue, Scripted | Lison & Tiedermann (2016) | [link](https://opus.nlpl.eu/OpenSubtitles-v2018.php) | Open source |
+| Simple English Wikipedia | 15% | Nonfiction | -- | [link](https://dumps.wikimedia.org/simplewiki/20221201/) | [link](https://dumps.wikimedia.org/legal.html) |
+| Switchboard | 1% | Dialogue | Godfrey et al. (1992), Stolcke et al., (2000) | [link](http://compprag.christopherpotts.net/swda.html) | [link](http://compprag.christopherpotts.net/swda.html) |
 
-## How to Get Started with the Model
+<sup>1</sup> Our distribution of part of the BNC Texts is permitted under the fair dealings provision of copyright law (see term (2g) in the BNC license).
 
-Use the code below to get started with the model.
+## Hyperparameters
 
-[More Information Needed]
+| Hyperparameter | Value |
+| --- | --- |
+| Number of epochs | 10 |
+| Datapoint length | 512 |
+| Batch size | 16 |
+| Learning rate | 0.00005 |
+| Number of steps | 200000 |
+| Warmup steps | 2000 |
+| Gradient clipping | 1 |
+| Optimizer | AdamW |
+| Optimizer Beta_1 | 0.9 |
+| Optimizer Beta_2 | 0.999 |
+| Optimizer Epsilon | 10<sup>-8</sup>|
 
-## Training Details
 
-### Training Data
+## Training Procedure
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+The model is trained with the next token prediction loss for 10 epochs.
 
-[More Information Needed]
+### Size and checkpoints
 
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+The model has 124M parameters.
+In total we train on around 100M words and provide multiple checkpoints from the training.
+Specifically we provode:
+- Checkpoints every 1M words for the first 10M words
+- Checkpoints every 10M words first 100M words
+ 
+# Evaluation
 
-#### Speeds, Sizes, Times [optional]
+<!-- This section describes the evaluation protocols and provides the results. -->
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+This model is evaluated in three different fashions:
+1. We do zero-shot evaluation on 7 tasks.
+2. We do fine-tuning on a subset of the (Super)GLUE tasks (Wang et al., ICLR 2019; Wang et al., NeurIPS 2019) .
 
-[More Information Needed]
+## Testing Data & Metrics
 
-## Evaluation
+### Testing Data
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+<!-- This should link to a Data Card if possible. -->
 
-### Testing Data, Factors & Metrics
+For the BLiMP, BLiMP supplement, and EWoK tasks, we use a filtered version of the dataset to only include examples with words found in the BabyLM dataset.
+For the Finetuning task, we both filter and sample down to a maximum 10 000 train examples.
 
-#### Testing Data
+*Validation Data*
 
-<!-- This should link to a Dataset Card if possible. -->
+*Zero-shot Tasks*
 
-[More Information Needed]
+- **BLiMP**: The Benchmark of Linguistic Minimal Pairs evaluates the model's linguistic ability by seeing if it can recognize the grammatically correct sentence from a pair of minimally different sentences. It tests various grammatical phenomena.(Warstadt et al., TACL 2020)
+- **BLiMP Supplement**: A supplement to BLiMP introduced in the first edition of the BabyLM challenge. More focused on dialogue and questions. (Warstadt et al., CoNLL-BabyLM 2023)
+- **EWoK**: Works similarly to BLiMP but looks the model's internal world knowledge. Looking at both whter a model has physical and social knowledge. (Ivanova et al., 2024)
+- **Eye Tracking and Self-paced Reading**: Looks at whether the model can mimick the eye tracking and reading time of a human but using surprisal of a word as a proxy for time spent reading a word. (de Varda et al., BRM 2024)
+- **Entity Tracking**: Checks whether a model can keep track of the changes to the states of entities as text/dialogue unfolds. (Kim & Schuster, ACL 2023)
+- **WUGs**: Tests morphological generalization in LMs through an adjective nominalization task. (Hofmann et al., 2024)
 
-#### Factors
+*Finetuning Tasks*
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+- **BoolQ**: A yes/no QA dataset with unprompted and unconstrained questions. (Clark et al., NAACL 2019)
+- **MNLI**: The Multi-Genre Natural Language Inference corpus tests the language understanding of a model by seeing wehther it can recognize textual entailment. (Williams et al., NAACL 2018)
+- **MRPC**: The Microsoft Research Paraphrase Corpus contains pairs of sentences that are either paraphrases/semntically equivalent to each other or unrelated.(Dolan & Brockett, IJCNLP 2005)
+- **QQP**<sup>2</sup>: Similarly to MRPC, the Quora Question Pairs corpus tests the models ability to determine whether a pair of questions are sematically similar to each other. These questions are sourced from Quora.
+- **MultiRC**: The Multi-Sentence Reading Comprehension corpus is a QA task that evaluates the model's ability to the correct answer from a list of answers given a question and context paragraph. In this version the data is changed to a binary classification judging whether the answer to a question, context pair is correct. (Khashabi et al., NAACL 2018)
+- **RTE**: Similar the Recognizing Text Entailement tests the model's ability to recognize text entailement. (Dagan et al., Springer 2006; Bar et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., TAC 2009)
+- **WSC**: The Winograd Schema Challenge tests the models ability to do coreference resolution on sentences with a pronoun and a list of noun phrases found in the sentence. This version edits it to be a binary classification on examples consisting of a pronoun and noun phrase.(Levesque et al., PKRR 2012)
 
-[More Information Needed]
+<sup>2</sup> https://www.quora.com/profile/Ricky-Riche-2/First-Quora-Dataset-Release-Question-Pairs
 
-#### Metrics
+### Metrics
 
 <!-- These are the evaluation metrics being used, ideally with a description of why. -->
 
-[More Information Needed]
-
-### Results
+The metrics used to evaluate the model are the following:
+- Zero-shot
+  - Accuracy on predicting the correct completion/sentence for BLiMP, BLiMP Supplement, EWoK, Entity Tracking, and WUGs
+  - Change in R^2 prediction from baseline for Eye Tracking (with no spillover) and Self-paced Reading (1-word spillover)
+- Finetuning
+  - 3 class Accuracy for MNLI
+  - Binary Accuracy for BoolQ, MultiRC, and WSC
+  - F1-score for MRPC and QQP
 
-[More Information Needed]
+The metrics were chosen based on the advice of the papers the tasks come from.
 
-#### Summary
+## Results 
 
+*Zero-shot*
 
+| Task | Metric | Causal Score | 
+| --- | --- | --- | 
+| BLiMP | Acc | 67.29 | 
+| BLiMP Supplement | Acc | 59.09 | 
+| EWoK | Acc | 49.8| 
+| Eye Tracking | change in R^2 | ADD |
+| Self-paced Reading | change in R^2 | ADD |
+| Entity Tracking | Acc | 18.92 | 
+| WUGs | Acc | 39 | 
 
-## Model Examination [optional]
+*Finetuning*
 
-<!-- Relevant interpretability work for the model goes here -->
+| Task | Metric | Uni-directional Score | Bi-directional Score |
+| --- | --- | --- | --- |
+| BoolQ | Acc | | |
+| MNLI | Acc | | |
+| MRPC | F1 | | |
+| QQP | F1 | | |
+| MultiRC | Acc | | |
+| RTE | Acc | | |
+| WSC | Acc | | |
 
-[More Information Needed]
+# Technical Specifications
 
-## Environmental Impact
+### Hardware
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+- 1 H100 GPU was used to train this model.
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+### Software
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+PyTorch
 
-## Technical Specifications [optional]
+### Training Time
 
-### Model Architecture and Objective
+The model took ~0.5 GPU hours to train.
 
-[More Information Needed]
+# Citation
 
-### Compute Infrastructure
+```latex
+@misc{charpentier2025babylmturns3papers,
+      title={BabyLM Turns 3: Call for papers for the 2025 BabyLM workshop}, 
+      author={Lucas Charpentier and Leshem Choshen and Ryan Cotterell and Mustafa Omer Gul and Michael Hu and Jaap Jumelet and Tal Linzen and Jing Liu and Aaron Mueller and Candace Ross and Raj Sanjay Shah and Alex Warstadt and Ethan Wilcox and Adina Williams},
+      year={2025},
+      eprint={2502.10645},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2502.10645}, 
+}
+```
 
-[More Information Needed]
+# Model Card Authors
 
-#### Hardware
+Mustafa Ömer Gül
 
-[More Information Needed]
+# Bibliography
 
-#### Software
+[GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://openreview.net/pdf?id=rJ4km2R5t7) (Wang et al., ICLR 2019)
 
-[More Information Needed]
+[SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://proceedings.neurips.cc/paper_files/paper/2019/file/4496bf24afe7fab6f046bf4923da8de6-Paper.pdf) (Wang et al., NeurIPS 2019)
 
-## Citation [optional]
+[BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://aclanthology.org/2020.tacl-1.25/) (Warstadt et al., TACL 2020)
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+[Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://aclanthology.org/2023.conll-babylm.1/) (Warstadt et al., CoNLL-BabyLM 2023)
 
-**BibTeX:**
+[🌏 Elements of World Knowledge (EWoK): A cognition-inspired framework for evaluating basic world knowledge in language models](https://arxiv.org/pdf/2405.09605v1) (Ivanova et al., 2024)
 
-[More Information Needed]
+[Cloze probability, predictability ratings, and computational estimates for 205 English sentences, aligned with existing EEG and reading time data](https://link.springer.com/article/10.3758/s13428-023-02261-8) (de Varda et al., BRM 2024)
 
-**APA:**
+[Entity Tracking in Language Models](https://aclanthology.org/2023.acl-long.213/) (Kim & Schuster, ACL 2023)
 
-[More Information Needed]
+[Derivational Morphology Reveals Analogical Generalization in Large Language Models](https://arxiv.org/pdf/2411.07990) (Hofmann et al., 2024)
 
-## Glossary [optional]
+[Automatically Constructing a Corpus of Sentential Paraphrases](https://aclanthology.org/I05-5002/) (Dolan & Brockett, IJCNLP 2005)
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+[A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://aclanthology.org/N18-1101/) (Williams et al., NAACL 2018)
 
-[More Information Needed]
+[The Winograd Schema Challenge]( http://dl.acm.org/citation.cfm?id=3031843.3031909) (Levesque et al., PKRR 2012)
 
-## More Information [optional]
+[The PASCAL Recognising Textual Entailment Challenge](https://link.springer.com/chapter/10.1007/11736790_9) (Dagan et al., Springer 2006)
 
-[More Information Needed]
+[The Second PASCAL Recognising Textual Entailment Challenge]() (Bar et al., 2006)
 
-## Model Card Authors [optional]
+[The Third PASCAL Recognizing Textual Entailment Challenge](https://aclanthology.org/W07-1401/) (Giampiccolo et al., 2007)
 
-[More Information Needed]
+[The Fifth PASCAL Recognizing Textual Entailment Challenge](https://tac.nist.gov/publications/2009/additional.papers/RTE5_overview.proceedings.pdf) (Bentivogli et al., TAC 2009)
 
-## Model Card Contact
+[BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://aclanthology.org/N19-1300/) (Clark et al., NAACL 2019)
 
-[More Information Needed]
+[Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences](https://aclanthology.org/N18-1023/) (Khashabi et al., NAACL 2018)