|
|
--- |
|
|
pipeline_tag: text-generation |
|
|
inference: false |
|
|
license: apache-2.0 |
|
|
library_name: transformers |
|
|
tags: |
|
|
- language |
|
|
- aquif |
|
|
- gpt2 |
|
|
- text-generation-inference |
|
|
- math |
|
|
- coding |
|
|
- small |
|
|
language: |
|
|
- en |
|
|
datasets: |
|
|
- facebook/empathetic_dialogues |
|
|
- openai/gsm8k |
|
|
- codeparrot/codeparrot-clean |
|
|
- brando/small-c4-dataset |
|
|
--- |
|
|
|
|
|
# aquif-neo |
|
|
|
|
|
**aquif-neo** is our first pretrained model, featuring 64.1 million parameters. Designed purely as an experiment, it currently does not yet offer coherent text and reasoning at all. |
|
|
|
|
|
## Model Overview |
|
|
- **Name**: `aquif-neo` |
|
|
- **Parameters**: 64.1 million |
|
|
- **Architecture**: Dense |
|
|
- **Type**: General-purpose LLM |
|
|
- **Hosted on**: [Hugging Face](https://huggingface.co/aquiffoo/aquif-neo) |
|
|
|
|
|
## Training Steps |
|
|
|
|
|
step 500 | loss = 0.9147 |
|
|
\ |
|
|
step 1000 | loss = 0.7440 |
|
|
\ |
|
|
step 1500 | loss = 0.6791 |
|
|
\ |
|
|
step 2000 | loss = 0.6631 |
|
|
\ |
|
|
step 2500 | loss = 0.6439 |
|
|
\ |
|
|
step 3000 | loss = 0.6335 |
|
|
\ |
|
|
step 3500 | loss = 0.6176 |
|
|
\ |
|
|
step 4000 | loss = 0.5987 |
|
|
\ |
|
|
step 4500 | loss = 0.5979 |
|
|
\ |
|
|
step 5000 | loss = 0.6018 |
|
|
\ |
|
|
step 5500 | loss = 0.5767 |
|
|
\ |
|
|
step 6000 | loss = 0.5839 |
|
|
\ |
|
|
step 6500 | loss = 0.5754 |
|
|
\ |
|
|
step 7000 | loss = 0.5644 |
|
|
\ |
|
|
step 7500 | loss = 0.5640 |
|
|
\ |
|
|
step 8000 | loss = 0.5686 |
|
|
|
