t5-v1_1-base-finetuned-English-to-BASH

Created by: Josh Shih, Alex Sha, Kevin Um for EEP 596 - Natural Language Processing at University of Washington (Seattle).

This model is a fine-tuned version of google/t5-v1_1-base on a more balanced iteration of the NL2BASH dataset. It achieves the following results on the evaluation set:

• Loss: 0.7958
• Nl2bash M: 0.6179
• Gen Len: 14.0821

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

This model was trained and evaluated using a custom iteration of NL2BASH. The original NL2BASH dataset contains a large class imbalance with too many bash commands which begin with 'find'.

A maximum threshold was set to remove text/BASH pairs which exceeded the threshold, and GPT-3 API was used to generate text/BASH pairs for those below the threshold.

~5500 original text/BASH pairs and ~5700 generated text/BASH pairs were used, giving a total of ~11200 lines of text/BASH pairs. Shown below is the class distribution for the top-5 commands.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss	Nl2bash M	Gen Len
4.7401	1.0	561	1.6374	0.258	12.9715
1.9636	2.0	1122	1.2574	0.345	14.4728
1.5349	3.0	1683	1.0844	0.3873	14.5727
1.3565	4.0	2244	0.9777	0.4705	14.3506
1.2006	5.0	2805	0.8995	0.5536	14.2159
1.0931	6.0	3366	0.8622	0.5848	14.0883
1.0008	7.0	3927	0.8261	0.603	14.1401
0.952	8.0	4488	0.8146	0.6145	13.9322
0.8829	9.0	5049	0.8013	0.6185	13.9715
0.8657	10.0	5610	0.7958	0.6179	14.0821

Framework versions

• Transformers 4.27.0.dev0
• Pytorch 1.13.1+cu116
• Datasets 2.10.0
• Tokenizers 0.13.2