metadata

license: apache-2.0
tags:
  - generated_from_trainer
datasets:
  - Multi-Domain-Expert-Layers/pubmed_central
metrics:
  - accuracy
model-index:
  - name: layer_9,10,11,12,13
    results:
      - task:
          type: text-generation
          name: Causal Language Modeling
        dataset:
          name: Multi-Domain-Expert-Layers/pubmed_central
          type: Multi-Domain-Expert-Layers/pubmed_central
          split: None
        metrics:
          - type: accuracy
            value: 0.5767534246575342
            name: Accuracy

layer_9,10,11,12,13

This model is a fine-tuned version of EleutherAI/pythia-1b-deduped on the Multi-Domain-Expert-Layers/pubmed_central dataset. It achieves the following results on the evaluation set:

Loss: 2.0227
Accuracy: 0.5768

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0001
train_batch_size: 1
eval_batch_size: 8
seed: 42
distributed_type: multi-GPU
num_devices: 8
gradient_accumulation_steps: 8
total_train_batch_size: 64
total_eval_batch_size: 64
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
training_steps: 1000

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
2.0567	0.0	200	2.0533	0.5717
2.041	0.01	400	2.0438	0.5733
2.0496	0.01	600	2.0361	0.5749
2.0194	0.02	800	2.0276	0.5761
2.0338	0.02	1000	2.0227	0.5768

Framework versions

Transformers 4.28.1
Pytorch 2.0.0+cu117
Datasets 2.11.0
Tokenizers 0.13.3

Wandb Report

https://wandb.ai/ontocord/pythia-1b-deduped-layer-test-pubmed_central/runs/yy3pwx0o