Commit
·
8dca0cc
1
Parent(s):
d7636f8
Update README.md with new details
Browse files
README.md
CHANGED
@@ -19,11 +19,11 @@ pipeline_tag: text-generation
|
|
19 |
|
20 |
# 🧪 Qwen2.5-0.5B-Instruct + LoRA Fine-Tuned on PubMedQA (pqa_labeled)
|
21 |
|
22 |
-
This model is a LoRA-adapted version of [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct), fine-tuned using [Unsloth](https://github.com/unslothai/unsloth) on the `pqa_labeled` subset of the [PubMedQA](https://huggingface.co/datasets/
|
23 |
|
24 |
## ✅ Summary
|
25 |
|
26 |
-
This work demonstrates that even a compact instruction-tuned model like Qwen2.5 0.5B can achieve near state-of-the-art performance on biomedical QA tasks. With LoRA fine-tuning
|
27 |
|
28 |
It reframes the classification task as a text generation problem — prompting the model to generate "yes", "no", or "maybe" responses. This results in highly interpretable and efficient predictions with excellent generalization.
|
29 |
|
@@ -59,10 +59,9 @@ It reframes the classification task as a text generation problem — prompting t
|
|
59 |
- `r`: 16
|
60 |
- `alpha`: 16
|
61 |
- `target_modules`: ["q_proj", "v_proj"]
|
62 |
-
- **Epochs:**
|
63 |
-
- **Batch Size:**
|
64 |
-
- **Learning Rate:**
|
65 |
-
- **NEFTune:** [Insert if used]
|
66 |
|
67 |
---
|
68 |
|
@@ -73,8 +72,8 @@ from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
73 |
from peft import PeftModel
|
74 |
|
75 |
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
|
76 |
-
model = PeftModel.from_pretrained(model, "ShahzebKhoso/qwen2.5-pubmedqa-lora")
|
77 |
-
tokenizer = AutoTokenizer.from_pretrained("ShahzebKhoso/qwen2.5-pubmedqa-lora")
|
78 |
```
|
79 |
|
80 |
---
|
@@ -94,7 +93,7 @@ tokenizer = AutoTokenizer.from_pretrained("ShahzebKhoso/qwen2.5-pubmedqa-lora")
|
|
94 |
title={Fine-tuning Qwen2.5-0.5B on PubMedQA with LoRA},
|
95 |
author={Shahzeb Khoso},
|
96 |
year={2025},
|
97 |
-
howpublished={\\url{https://huggingface.co/ShahzebKhoso/qwen2.5-pubmedqa-lora}},
|
98 |
}
|
99 |
```
|
100 |
|
@@ -103,6 +102,6 @@ tokenizer = AutoTokenizer.from_pretrained("ShahzebKhoso/qwen2.5-pubmedqa-lora")
|
|
103 |
## ✨ Acknowledgements
|
104 |
|
105 |
- [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
|
106 |
-
- [PubMedQA Dataset](https://huggingface.co/datasets/
|
107 |
- [Unsloth](https://github.com/unslothai/unsloth)
|
108 |
- [Hugging Face PEFT](https://github.com/huggingface/peft)
|
|
|
19 |
|
20 |
# 🧪 Qwen2.5-0.5B-Instruct + LoRA Fine-Tuned on PubMedQA (pqa_labeled)
|
21 |
|
22 |
+
This model is a LoRA-adapted version of [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct), fine-tuned using [Unsloth](https://github.com/unslothai/unsloth) on the `pqa_labeled` subset of the [PubMedQA](https://huggingface.co/datasets/qiaojin/PubMedQA) dataset.
|
23 |
|
24 |
## ✅ Summary
|
25 |
|
26 |
+
This work demonstrates that even a compact instruction-tuned model like Qwen2.5 0.5B Instruct can achieve near state-of-the-art performance on biomedical QA tasks. With LoRA fine-tuning using just 1,000 examples, this model achieves **98.99% accuracy** on the PubMedQA test set.
|
27 |
|
28 |
It reframes the classification task as a text generation problem — prompting the model to generate "yes", "no", or "maybe" responses. This results in highly interpretable and efficient predictions with excellent generalization.
|
29 |
|
|
|
59 |
- `r`: 16
|
60 |
- `alpha`: 16
|
61 |
- `target_modules`: ["q_proj", "v_proj"]
|
62 |
+
- **Epochs:** 100
|
63 |
+
- **Batch Size:** 16
|
64 |
+
- **Learning Rate:** 2e-4
|
|
|
65 |
|
66 |
---
|
67 |
|
|
|
72 |
from peft import PeftModel
|
73 |
|
74 |
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
|
75 |
+
model = PeftModel.from_pretrained(model, "ShahzebKhoso/qwen2.5-instruct-0.5B-pubmedqa-lora")
|
76 |
+
tokenizer = AutoTokenizer.from_pretrained("ShahzebKhoso/qwen2.5-instruct-0.5B-pubmedqa-lora")
|
77 |
```
|
78 |
|
79 |
---
|
|
|
93 |
title={Fine-tuning Qwen2.5-0.5B on PubMedQA with LoRA},
|
94 |
author={Shahzeb Khoso},
|
95 |
year={2025},
|
96 |
+
howpublished={\\url{https://huggingface.co/ShahzebKhoso/qwen2.5-instruct-0.5B-pubmedqa-lora}},
|
97 |
}
|
98 |
```
|
99 |
|
|
|
102 |
## ✨ Acknowledgements
|
103 |
|
104 |
- [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
|
105 |
+
- [PubMedQA Dataset](https://huggingface.co/datasets/qiaojin/PubMedQA)
|
106 |
- [Unsloth](https://github.com/unslothai/unsloth)
|
107 |
- [Hugging Face PEFT](https://github.com/huggingface/peft)
|