Create README.md

README.md

@@ -0,0 +1,118 @@
+# ---
+license: apache-2.0
+datasets:
+- common-pile/comma-dataset
+language:
+- en
+tags:
+- openly-licensed
+- llm
+- pretraining
+---
+
+# Model Card for Model ID
+
+Comma v0.1 is a 7 billion parameter model trained on 1 trillion tokens of openly licensed text collected as part of the Common Pile.
+
+## Model Details
+
+### Model Description
+
+Comma v0.1 is a 7 billion parameter decoder-only transformer. It uses the same architecture as Llama 3. It was trained on 1 trillion tokens from the Common Pile, an 8TB collection of openly licensed text.
+
+- **Developed by:** r-three, Eulther AI, Vector, University of Toronto
+<!-- - **Funded by [optional]:** [More Information Needed] -->
+- **Model type:** Decoder-Only Transformer
+- **Language(s) (NLP):** English
+- **License:** Apache 2.0
+
+### Model Sources
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** https://github.com/r-three/common-pile/
+- **Paper:** In Progress
+
+## Uses
+
+Comma v0.1 can be used a the starting point for finetuning and post-training. As it was trained on openly licensed text, it is less likely to create IP issues, but this is not a guarantee.
+
+### Direct Use
+
+Evaluations in our [paper]() show performance when using our final model directly. Additional post-training will mostly likely increase performance.
+
+### Out-of-Scope Use
+
+Comma v0.1 is only trained on openly licensed text. Therefore it will probably have reduced performance when asked about topics that only appear in copyrighted text.
+
+## Bias, Risks, and Limitations
+
+As it was trained on openly licensed text, Comma v0.1 is less likely to output IP infringing text, however, due to issues like license laundering this is not a guarantee. See our [paper]() for a deeper discussion of these details. 
+
+Comma v0.1 is trained on many old books (pre 1929) and may therefore repeat societal biases common at the time.
+
+Comma v0.1 include no post-hoc guardrails that limit what it may generate.
+
+### Recommendations
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## Training Details
+
+### Training Data
+
+Comma v0.1 was train on the [common-pile/comma-data](https://huggingface.co/datasets/common-pile/comma-dataset), a filtered and deduplicted dataset of 1 trillion tokens drawn from the [Common Pile](https://huggingface.co/collections/common-pile/common-pile-v01-6826b454a5a6a445d0b51b37) dataset (8TB of openly licensed text).
+
+### Training Procedure
+
+Comma v0.1 was trained in two stages, first it was trained for 965 billion tokens with a cosine learning rate. Then a second "cool-down" training phase on 35 billion tokens from high quality sources was done. The final model is the average of 10 checkpoints during this cool-down phase.
+
+#### Training Hyperparameters
+
+Hyperparameters can be found in our [lingua config file](https://huggingface.co/common-pile/comma-v0.1-checkpoints/blob/main/config.yaml).
+
+## Evaluation
+
+Comma v1.0 7B outperforms models with similar computational budgets (7 billion parameters, 1 trillion tokens) that were trained on non-openly licensed text (LLaMA 1, MPT, RPJ-INCITE) on several common benchmarks (ARC-C, MMLU, BoolQ, SIQA etc.) and does especially well on Code based tasks (HumEval, MBPP). It tends to under performs on datasets like HellaSwag. Evaluations where done using OLMES. Note that there is still a large gap between Comma v0.1 and current state-of-the-art models line Qwen3 which was trained on 36 times as many tokens.
+
+More evaluation results can be found in our [paper]()
+
+#### Summary
+
+Comma v0.1 is a 7B parameter model train on openly licensed text. It is one of the first performant model trained on **only** open licensed text.
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** H100 Nvidia GPU
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** AWS
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications
+
+### Model Architecture and Objective
+
+Comma v0.1 uses the same architecture as Llama 3 and is trained using standard autoregressive next-token prediction.
+
+### Compute Infrastructure
+
+Comma v0.1 was trained on the Huggingface Cluster.
+
+#### Hardware
+
+Comma v0.1 was trained using 64 H100 Nvidia GPUs
+
+#### Software
+
+Comma v0.1 was trained using [lingua](https://github.com/facebookresearch/lingua)
+
+## Citation
+
+```bibtext
+
+```