gollm-12.8b-instruct-v2.3

This model is a fine-tuned version of EleutherAI/polyglot-ko-12.8b on a custom mixed dataset

Model description

• No-context template

아래는 작업을 설명하는 질문어와 추가 컨텍스트를 제공하는 맥락이 함께 제공됩니다. 요청을 적절히 완료하는 답변을 작성하세요.

### 질문:
{instruction}

### 답변:

• With context template

아래는 작업을 설명하는 질문어와 추가 컨텍스트를 제공하는 맥락이 함께 제공됩니다. 요청을 적절히 완료하는 답변을 작성하세요.

### 맥락:
{input}

### 질문:
{instruction}

### 답변:

Intended uses & limitations

More information needed

Training and evaluation data

• self-introduction (20 samples)
• High-quality reasoning dataset from private documents, QAs generated by Claude AI (1.3k samples)
• EverythingLM-v2 (0.9k samples)
• KoCoT (2k samples)
• Private MRC dataset - answer generated by GPT-4 (32k samples) Original data have ~12k question-answer pairs with context, and augmentation is applied to make 20k samples with triplet contexts case (1 correct context out of 3)

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 2
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 8
• saved_checkpoint_at_epoch: 1 (condition: loss < 0.3)

Framework versions

• Transformers 4.32.0.dev0
• Pytorch 2.0.0+cu117
• Datasets 2.11.0
• Tokenizers 0.13.3