|
|
--- |
|
|
library_name: transformers |
|
|
tags: [] |
|
|
--- |
|
|
--- |
|
|
# phi2-2b-absa: Fine-Tuned Aspect-Based Sentiment Analysis Model |
|
|
|
|
|
## Model Description |
|
|
|
|
|
The **phi2-2b-absa** model is a fine-tuned aspect-based sentiment analysis (ABSA) model based on the Microsoft Phi-2 model. It has been trained on the **semeval2016-full-absa-reviews-english-translated-resampled** dataset. The model predicts sentiments towards different aspects mentioned in a given sentence. |
|
|
|
|
|
## Fine-Tuning Details |
|
|
|
|
|
The fine tuning can be revisited on [Google Colab](https://colab.research.google.com/drive/1n3ykETLpHQPXwPhUcOe-z9cG3ThrDkSi?usp=sharing). |
|
|
|
|
|
### Dataset |
|
|
- **Name:** semeval2016-full-absa-reviews-english-translated-resampled |
|
|
- **Description:** Annotated dataset for ABSA containing sentences, aspects, sentiments, and additional contextual text. It is split into train and test sets. |
|
|
|
|
|
### Model Architecture |
|
|
- **Base Model:** Microsoft Phi-2 |
|
|
- **Fine-Tuned Model:** phi2-2b-absa |
|
|
|
|
|
### Fine-Tuning Parameters |
|
|
- **LoRA Attention Dimension (lora_r):** 64 |
|
|
- **LoRA Scaling Parameter (lora_alpha):** 16 |
|
|
- **LoRA Dropout Probability (lora_dropout):** 0.1 |
|
|
|
|
|
### BitsAndBytes Quantization |
|
|
- **Activate 4-bit Precision:** True |
|
|
- **Compute Dtype for 4-bit Models:** float16 |
|
|
- **Quantization Type:** nf4 |
|
|
|
|
|
### Training Parameters |
|
|
- **Number of Training Epochs:** 1 |
|
|
- **Batch Size per GPU for Training:** 4 |
|
|
- **Batch Size per GPU for Evaluation:** 4 |
|
|
- **Gradient Accumulation Steps:** 1 |
|
|
- **Learning Rate:** 2e-4 |
|
|
- **Weight Decay:** 0.001 |
|
|
- **Optimizer:** PagedAdamW (32-bit) |
|
|
- **Learning Rate Scheduler:** Cosine |
|
|
|
|
|
### SFT Parameters |
|
|
- **Maximum Sequence Length:** None |
|
|
- **Packing:** False |
|
|
|
|
|
## How to Use |
|
|
|
|
|
``` |
|
|
from transformers import AutoTokenizer, pipeline |
|
|
import torch |
|
|
|
|
|
model = "Alpaca69B/llama-2-7b-absa-semeval-2016" |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model) |
|
|
pipeline = pipeline( |
|
|
"text-generation", |
|
|
model=model, |
|
|
tokenizer=tokenizer, |
|
|
torch_dtype=torch.float16, |
|
|
device="auto", |
|
|
) |
|
|
|
|
|
input_sentence = "the first thing that attracts attention is the warm reception and the smiling receptionists." |
|
|
sequences = pipeline( |
|
|
f'### Human: {input_sentence} ### Assistant: aspect:', |
|
|
do_sample=True, |
|
|
top_k=10, |
|
|
num_return_sequences=1, |
|
|
eos_token_id=tokenizer.eos_token_id, |
|
|
max_length=200, |
|
|
) |
|
|
sequences[0]['generated_text'] |
|
|
|
|
|
``` |
|
|
|
|
|
Testing can be seen on [Google Colab](https://colab.research.google.com/drive/1eKdZYYWiivyeCQDsocGBstVODMLZyT-_?usp=sharing) |
|
|
|
|
|
## Acknowledgments |
|
|
|
|
|
- The fine-tuning process and model development were performed by Ben Kampmann. |
|
|
--- |
