Consolidate README

README.md

@@ -1,118 +1,51 @@
 # ---
 license: apache-2.0
 datasets:
-- common-pile/comma-dataset
+- common-pile/comma_v0.1_training_dataset
 language:
 - en
-tags:
-- openly-licensed
-- llm
-- pretraining
 ---
 
-# Model Card for Model ID
+# Comma v0.1
 
-Comma v0.1 is a 7 billion parameter model trained on 1 trillion tokens of openly licensed text collected as part of the Common Pile.
+Comma v0.1 is a 7 billion parameter language model trained on [the Comma v0.1 dataset](https://huggingface.co/datasets/common-pile/comma_v0.1_training_dataset), comprising 1 trillion tokens of openly licensed text from [the Common Pile](https://huggingface.co/collections/common-pile/common-pile-v01-68307d37df48e36f02717f21).
+Comma v0.1 is a "base model" that can be used a the starting point for finetuning and post-training.
+It performs comparably to budget-matched models (7 billion parameters, 1 trillion tokens) trained on unlicensed data.
 
-## Model Details
+| Model      | ARC-C | ARC-E | MMLU | BoolQ | HSwag | OBQA | CSQA | PIQA | SIQA | HEval | MBPP | Avg. |
+| ---------- | ----- | ----- | ---- | ----- | ----- | ---- | ---- | ---- | ---- | ----- | ---- | ---- |
+| RPJ-INCITE | 42.8  | 68.4  | 27.8 | 68.6  | 70.3  | 49.4 | 57.7 | 76   | 46.9 | 11.1  | 15.9 | 48.6 |
+| LLaMA 1    | 44.5  | 67.9  | 34.8 | 75.4  | 76.2  | 51.2 | 61.8 | 77.2 | 50.3 | 19.9  | 27.9 | 53.4 |
+| StableLM   | 50.8  | 65.4  | 45.2 | 71.7  | 75.6  | 48.2 | 57.2 | 77   | 48.2 | 23.1  | 32   | 54.0 |
+| MPT        | 46.5  | 70.5  | 30.2 | 74.2  | 77.6  | 48.6 | 63.3 | 77.3 | 49.1 | 27.3  | 33.2 | 54.3 |
+| OpenLLaMA  | 44.5  | 67.2  | 40.3 | 72.6  | 72.6  | 50.8 | 62.8 | 78   | 49.7 | 27.6  | 33.9 | 54.5 |
+| Comma v0.1 | 52.8  | 68.4  | 42.4 | 75.7  | 62.6  | 47   | 59.4 | 70.8 | 50.8 | 36.5  | 35.5 | 54.7 |
 
-### Model Description
-
-Comma v0.1 is a 7 billion parameter decoder-only transformer. It uses the same architecture as Llama 3. It was trained on 1 trillion tokens from the Common Pile, an 8TB collection of openly licensed text.
-
-- **Developed by:** r-three, Eulther AI, Vector, University of Toronto
-<!-- - **Funded by [optional]:** [More Information Needed] -->
-- **Model type:** Decoder-Only Transformer
-- **Language(s) (NLP):** English
-- **License:** Apache 2.0
-
-### Model Sources
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** https://github.com/r-three/common-pile/
-- **Paper:** In Progress
-
-## Uses
-
-Comma v0.1 can be used a the starting point for finetuning and post-training. As it was trained on openly licensed text, it is less likely to create IP issues, but this is not a guarantee.
-
-### Direct Use
-
-Evaluations in our [paper]() show performance when using our final model directly. Additional post-training will mostly likely increase performance.
-
-### Out-of-Scope Use
-
-Comma v0.1 is only trained on openly licensed text. Therefore it will probably have reduced performance when asked about topics that only appear in copyrighted text.
-
-## Bias, Risks, and Limitations
-
-As it was trained on openly licensed text, Comma v0.1 is less likely to output IP infringing text, however, due to issues like license laundering this is not a guarantee. See our [paper]() for a deeper discussion of these details. 
-
-Comma v0.1 is trained on many old books (pre 1929) and may therefore repeat societal biases common at the time.
-
-Comma v0.1 include no post-hoc guardrails that limit what it may generate.
-
-### Recommendations
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## Training Details
-
-### Training Data
-
-Comma v0.1 was train on the [common-pile/comma-data](https://huggingface.co/datasets/common-pile/comma-dataset), a filtered and deduplicted dataset of 1 trillion tokens drawn from the [Common Pile](https://huggingface.co/collections/common-pile/common-pile-v01-6826b454a5a6a445d0b51b37) dataset (8TB of openly licensed text).
-
-### Training Procedure
-
-Comma v0.1 was trained in two stages, first it was trained for 965 billion tokens with a cosine learning rate. Then a second "cool-down" training phase on 35 billion tokens from high quality sources was done. The final model is the average of 10 checkpoints during this cool-down phase.
-
-#### Training Hyperparameters
+## Training details
 
+Comma v0.1 is a decoder-only transformer that uses the same architecture as Llama 3.
+Training was done in two stages: first on 965 billion tokens with a cosine learning rate schedule, and second a "cool-down" training phase on 35 billion tokens from high-quality sources.
+The final model is the average of 10 checkpoints during this cool-down phase.
+Training was performed using [https://github.com/facebookresearch/lingua/](lingua) on 64 Nvidia H100 GPUs.
 Hyperparameters can be found in our [lingua config file](https://huggingface.co/common-pile/comma-v0.1-checkpoints/blob/main/config.yaml).
 
-## Evaluation
-
-Comma v1.0 7B outperforms models with similar computational budgets (7 billion parameters, 1 trillion tokens) that were trained on non-openly licensed text (LLaMA 1, MPT, RPJ-INCITE) on several common benchmarks (ARC-C, MMLU, BoolQ, SIQA etc.) and does especially well on Code based tasks (HumEval, MBPP). It tends to under performs on datasets like HellaSwag. Evaluations where done using OLMES. Note that there is still a large gap between Comma v0.1 and current state-of-the-art models line Qwen3 which was trained on 36 times as many tokens.
-
-More evaluation results can be found in our [paper]()
-
-#### Summary
-
-Comma v0.1 is a 7B parameter model train on openly licensed text. It is one of the first performant model trained on **only** open licensed text.
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+## Limitations
 
-- **Hardware Type:** H100 Nvidia GPU
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** AWS
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications
-
-### Model Architecture and Objective
-
-Comma v0.1 uses the same architecture as Llama 3 and is trained using standard autoregressive next-token prediction.
-
-### Compute Infrastructure
-
-Comma v0.1 was trained on the Huggingface Cluster.
-
-#### Hardware
-
-Comma v0.1 was trained using 64 H100 Nvidia GPUs
-
-#### Software
-
-Comma v0.1 was trained using [lingua](https://github.com/facebookresearch/lingua)
+Comma v0.1 was trained only on English-language data and code from the 15 programming languages covered by the [stack-edu classifiers](https://huggingface.co/collections/HuggingFaceTB/the-ultimate-collection-of-code-classifiers-67b5aa3eb8994a4b71453005).
+It will likely have poor performance on other languages or programming languages.
+While we aimed to train solely on openly licensed data, license laundering and inaccurate metadata can result in erroneous license information in the Common Pile (for further discussion of this limitation, please see [our paper](TODO link)).
+Consequently, we cannot make a guarantee that Comma v0.1 was trained exclusively on openly licensed text.
+When preparing Comma v0.1's pre-training data, we made use of the Toxicity tagger from [Dolma](https://github.com/allenai/dolma) to attempt to remove problematic content.
+However, Comma v0.1 may nevertheless reflect social biases present in its training data.
+Finally, please note that Comma v0.1 is a base model that has not undergone any form of "alignment" and therefore has no guardrails that limit what it may generate.
 
 ## Citation
 
 ```bibtext
-
-```
+@article{kandpal2025common,
+  title={{The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text}},
+  author={Nikhil Kandpal and Brian Lester and Colin Raffel and Sebastian Majstorovic and Stella Biderman and Baber Abbasi and Luca Soldaini and Enrico Shippole and A. Feder Cooper and Aviya Skowron and Shayne Longpre and Lintang Sutawika and Alon Albalak and Zhenlin Xu and Guilherme Penedo and Loubna Ben  and Elie Bakouch and John David  and Honglu Fan and Dashiell Stander and Guangyu Song and Aaron Gokaslan and John Kirchenbauer and Tom Goldstein and Brian R and Bhavya Kailkhura and Tyler Murray},
+  journal={arXiv preprint},
+  year={2025}
+}
+```