File size: 1,581 Bytes
f485e36 a325a02 f485e36 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
## A Lossless Syntax Tree Generator with Zero-shot Error Correction
- We follow [jam](https://huggingface.co/apcl/jam)'s pretraining procedure and use the same data to pretrain except we also use srcml to pretrain the models.
- In the finetuning stage, we finetune our models for 3 epochs.
- Our [GitHub repo](https://github.com/apcl-research/autorepair) contains the code for reproduction using the same [data](https://huggingface.co/datasets/apcl/autorepair).
## Pretrained model parameters
| Hyperparameter | Description | Value |
| ----------- | ----------- |------------|
|e | embedding dimensions | 1024 |
|L | number of layers | 24 |
|h | attention heads | 16 |
|c | block size / context length | 256 |
|b | batch size | 4 |
|a | accumulation steps | 32 |
|r | learning rate | 3e-5 |
|y | weight decay | 1e-5 |
|iter | iterations | 570000 |
## Model files
| Filename | Description |
| ------- | ------- |
|ckpt.pt|A model file for finetuning|
|ckpt_base.pt | A model file for generating syntax tree with the error correction in zero-shot setting|
|ckpt_finetune.pt | A model finetuned with the syntatic error dataset |
- Note that you can adjust the batch size and accumulation steps based on your GPU memory. But, the batch size * accumulation steps should be 128.
- If you finetune your models with multiple GPUs, you can turn down accumulation steps. For example, if you finetune with 2 GPUs, you will need to half the accumulation steps.
|