simple-distilbert-model

This model is a fine-tuned version of distilbert-base-uncased on a simple customer and client conversational amazon dataset.

Model description

This model is developed for the research purpose, and the weights are updated at the attention layers with very fewer dataset.

Intended uses & limitations

main intend of this model is to behave as product management and compliance emotional classification model. This is just an initiative level model, not so well suitable for any purpose. less efficient (jus an initiative model).

Training and evaluation data

trained with only fewer amazons customer client conversation dataset(5000 entries) validation data with(1000 entries)

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 64
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	63	0.5133

Framework versions

• Transformers 4.48.2
• Pytorch 2.6.0+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0