---
library_name: transformers
language:
- en
license: apache-2.0
base_model: BEE-spoke-data/tFINE-680m-e32-d16-gqa-1024
tags:
- flan
- t5
- gqa
- instruct
datasets:
- pszemraj/flan-subsets-deduped
---


# tFINE-680m-e32-d16-gqa-flan

FLAN-tuned variant of a tFINE (t5) model with GQA.

- 32 encoder layers
- 16 decoder layers
- 1024 hidden size

## testing 


install [transformers fork with GQA updates for t5](https://github.com/pszemraj/transformers/tree/t5-gqa) (⚠️WIP🚧):

```sh
pip install -U git+https://github.com/pszemraj/transformers.git@t5-gqa
```

then 

```py
# pip install -U git+https://github.com/pszemraj/transformers.git@t5-gqa
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("BEE-spoke-data/tFINE-680m-e32-d16-gqa-flan")
model = AutoModelForSeq2SeqLM.from_pretrained(
    "BEE-spoke-data/tFINE-680m-e32-d16-gqa-flan"
)

prompt = "What is the capital of France?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=64, no_repeat_ngram_size=3)
print(
    tokenizer.batch_decode(
        generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True
    )[0]
)
```

## Quick eval

Quick eval for:	`BEE-spoke-data/tFINE-680m-e32-d16-gqa-flan`


hf (pretrained=BEE-spoke-data/tFINE-680m-e32-d16-gqa-flan,trust_remote_code=True,dtype=bfloat16,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8

|    Tasks    |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-------------|------:|------|-----:|--------|---|-----:|---|------|
|boolq        |      2|none  |     0|acc     |↑  |0.7040|±  |0.0080|
|openbookqa   |      1|none  |     0|acc     |↑  |0.1580|±  |0.0163|
|             |       |none  |     0|acc_norm|↑  |0.2420|±  |0.0192|
|piqa         |      1|none  |     0|acc     |↑  |0.6132|±  |0.0114|
|             |       |none  |     0|acc_norm|↑  |0.6159|±  |0.0113|
|social_iqa   |      0|none  |     0|acc     |↑  |0.4319|±  |0.0112|
|tinyArc      |      0|none  |    25|acc_norm|↑  |0.2898|±  |   N/A|
|tinyHellaswag|      0|none  |    10|acc_norm|↑  |0.3295|±  |   N/A|
|tinyMMLU     |      0|none  |     0|acc_norm|↑  |0.2980|±  |   N/A|
|winogrande   |      1|none  |     0|acc     |↑  |0.5020|±  |0.0141|

## Training and evaluation data

used config 'all'

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 8e-05
- train_batch_size: 4
- eval_batch_size: 2
- seed: 17868
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 32
- total_train_batch_size: 256
- total_eval_batch_size: 4
- optimizer: Use paged_ademamix_32bit and the args are:
No additional optimizer arguments
- lr_scheduler_type: constant_with_warmup
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 1.0