File size: 2,773 Bytes
129a662 4ade264 129a662 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 |
---
license: mit
datasets:
- pacovaldez/stackoverflow-questions
language:
- en
base_model:
- google/flan-t5-base
tokenuzer:
- google/flan-t5-base
library_name: transformers
tags:
- Stackoverflow
- flan-t5
- peft
- lora
- seq2seq
---
# 🤖 FLAN-T5 Base Fine-Tuned on Stack Overflow Questions (LoRA)
This is a fine-tuned version of [`google/flan-t5-base`](https://huggingface.co/google/flan-t5-base) on a curated dataset of Stack Overflow programming questions. It was trained using [LoRA](https://arxiv.org/abs/2106.09685) (Low-Rank Adaptation) for parameter-efficient fine-tuning, making it compact, efficient, and effective at modeling developer-style Q&A tasks.
---
## 🧠 Model Objective
The model is trained to:
- Rewrite or improve unclear programming questions
- Generate relevant clarifying questions or answers
- Summarize long developer queries
- Serve as a code-aware Q&A assistant
---
## 📚 Training Data
- **Source**: Stack Overflow public questions dataset (cleaned)
- **Format**: Instruction-like examples, Q&A pairs, summarization prompts
- **Cleaning**: HTML stripping, markdown-to-text, code-preserved
- **Size**: ~15k examples
---
## 🏗️ Training Details
- **Base Model**: `google/flan-t5-base`
- **Adapter Format**: LoRA using [`peft`](https://github.com/huggingface/peft)
- **Files**:
- `adapter_model.safetensors`
- `adapter_config.json`
- **Hyperparameters**:
- `r`: 8
- `lora_alpha`: 16
- `lora_dropout`: 0.1
- `bias`: "none"
- `task_type`: "SEQ_2_SEQ_LM"
- **Inference Mode**: Enabled
---
## 💡 How to Use
```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel
# Load tokenizer and base model
tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-base")
base_model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base")
# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "your-model-folder")
model.eval()
# Inference
prompt = "Rewrite this question more clearly: why is my javascript function undefined?"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
🧪 Intended Use
This model is best suited for:
Code-aware chatbot assistants
Prompt engineering for developer tools
Developer-focused summarization / rephrasing
Auto-moderation / clarification of tech questions
⚠️ Limitations
Not trained for code generation or long-form answers
May hallucinate incorrect or generic responses
Finetuned only on Stack Overflow — domain-specific
✨ Acknowledgements
Hugging Face Transformers
LoRA (PEFT)
Stack Overflow for open data
FLAN-T5: Scaling Instruction-Finetuned Models
🛠️ Created with love by Kunj | Model suggestion & guidance by ChatGPT |