giskardai/giskard-evaluator · Report for bhadresh-savani/distilbert-base-uncased-emotion

Hi Team,

This is a report from Giskard Bot Scan 🐢.

We have identified 2 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset dair-ai/emotion (subset split, split validation).

👉Performance issues (1)

For records in the dataset where text contains "know", the Precision is 5.25% lower than the global Precision.

Level	Data slice	Metric	Deviation
medium 🟡	`text` contains "know"	Precision = 0.885	-5.25% than global

Taxonomy

avid-effect:performance:P0204

🔍✨Examples

	text	label	Predicted `label`
17	i know what it feels like he stressed glaring down at her as she squeezed more soap onto her sponge	anger	sadness (p = 0.98)
91	i feel like the people i know are really generous and i have my needs met	joy	love (p = 0.80)
164	i have stayed at heritage christian because of the fulfillment that i feel in doing christ s work in action by being the hands the eyes the legs and the voice of supporting the individuals that i have been blessed to know and support	joy	love (p = 0.82)

👉Robustness issues (1)

When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 22.2% of the cases. We expected the predictions not to be affected by this transformation.

Level	Data slice	Metric	Deviation
major 🔴	—	Fail rate = 0.222	222/1000 tested samples (22.2%) changed prediction after perturbation

Taxonomy

avid-effect:performance:P0201

🔍✨Examples

	text	Add typos(text)	Original prediction	Prediction after perturbation
656	i feel a little bit more nostalgic when those memories come to mind	i feel a little bit more nosftalic when those memories comwe to mind	love (p = 1.00)	joy (p = 0.99)
734	i can talk to her about almost anything i want to and she just listens and she doesnt make me feel like a whiney brat and she helps me sort my thoughts and make decisions while keeping me where she feels im safe	i can talk to her about almost anything i want to and she just lisrens and she doesnt make me feel liek a shiney brat and she helps me sort my thoughts and make decisions while keeping me where she fes im safe	sadness (p = 1.00)	joy (p = 1.00)
1403	i feel the need to preface this by saying that i am strongly in favor of keeping violent or otherwise inappropriate videogames out of the hands of minors and i believe that this is an issue that parents and the government need to work on together	i feel the need to preface this by saying that i am ateongly in faor of keeping volent or otherwise inappropriate videogames outo f yhe hands of minor san di believe that this is an issue that parents and the government need to work on tovether	anger (p = 1.00)	sadness (p = 0.88)

Checkout out the Giskard Space and test your model.

Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.