Update from earendil

README.md

@@ -0,0 +1,101 @@
+---
+language: el
+pipeline_tag: zero-shot-classification
+tags:
+- xlm-roberta-base
+datasets:
+- multi_nli
+- snli
+- allnli_greek
+metrics:
+- accuracy
+license: apache-2.0
+widget:
+- text: "Το Facebook κυκλοφόρησε τα πρώτα «έξυπνα» γυαλιά επαυξημένης πραγματικότητας"
+  candidate_labels: "πολιτική, τεχνολογία, αθλητισμός"
+---
+
+# Cross-Encoder for Greek Natural Language Inference (Textual Entailment) & Zero-Shot Classification 
+This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.
+#### By the
+## Training Data
+The model was trained on the the Greek version of the combined AllNLI dataset([SNLI](https://nlp.stanford.edu/projects/snli/) and [MultiNLI](https://cims.nyu.edu/~sbowman/multinli/)) which was created using EN2EL NMT model available [here](https://huggingface.co/lighteternal/SSE-TUC-mt-en-el-cased).
+
+The model can be used in two ways:
+* NLI/Textual Entailment: For a given sentence pair, it will output three scores corresponding to the labels: contradiction, entailment, neutral.
+* Zero-shot classification through the Huggingface pipeline: Given a sentence and a set of labels/topics, it will output the likelihood of the sentence belonging to each of the topic. Under the hood, the logit for entailment between the sentence and each label is taken as the logit for the candidate label being valid.
+
+## Performance
+
+Evaluation on classification accuracy (entailment, contradiction, neutral) on mixed (Greek+English) AllNLI-dev set:
+| Metric | Value |
+| --- | --- |
+| Accuracy | 0.8409 |
+
+
+
+## To use the model for NLI/Textual Entailment
+
+#### Usage with sentence_transformers
+
+Pre-trained models can be used like this:
+```python
+from sentence_transformers import CrossEncoder
+model = CrossEncoder('MODEL_NAME')
+scores = model.predict([('Δύο άνθρωποι συναντιούνται στο δρόμο', 'Ο δρόμος έχει κόσμο'),
+                        ('Ένα μαύρο αυτοκίνητο ξεκινάει στη μέση του πλήθους.', 'Ένας άντρας οδηγάει σε ένα μοναχικό δρόμο'), 
+                        ('Δυο γυναίκες μιλάνε στο κινητό', 'Το τραπέζι ήταν πράσινο')])
+
+
+#Convert scores to labels
+label_mapping = ['contradiction', 'entailment', 'neutral']
+labels = [label_mapping[score_max] for score_max in scores.argmax(axis=1)]
+print(scores, labels)
+
+# Οutputs
+#[[-3.1526504  2.9981945 -0.3108107]
+# [ 5.0549307 -2.757949  -1.6220676]
+# [-0.5124733 -2.2671669  3.1630592]] ['entailment', 'contradiction', 'neutral']
+ ```
+ 
+#### Usage with Transformers AutoModel
+You can use the model also directly with Transformers library (without SentenceTransformers library):
+```python
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+import torch
+
+model = AutoModelForSequenceClassification.from_pretrained('MODEL_NAME')
+tokenizer = AutoTokenizer.from_pretrained('MODEL_NAME')
+
+features = tokenizer(['Δύο άνθρωποι συναντιούνται στο δρόμο', 'Ο δρόμος έχει κόσμο'], 
+                    ['Ένα μαύρο αυτοκίνητο ξεκινάει στη μέση του πλήθους.', 'Ένας άντρας οδηγάει σε ένα μοναχικό δρόμο.'],  
+                    padding=True, truncation=True, return_tensors="pt")
+
+model.eval()
+with torch.no_grad():
+    scores = model(**features).logits
+    label_mapping = ['contradiction', 'entailment', 'neutral']
+    labels = [label_mapping[score_max] for score_max in scores.argmax(dim=1)]
+    print(labels)
+```
+
+## To use the model for Zero-Shot Classification
+This model can also be used for zero-shot-classification:
+```python
+from transformers import pipeline
+
+classifier = pipeline("zero-shot-classification", model='MODEL_NAME')
+
+sent = "Το Facebook κυκλοφόρησε τα πρώτα «έξυπνα» γυαλιά επαυξημένης πραγματικότητας"
+candidate_labels = ["πολιτική", "τεχνολογία", "αθλητισμός"]
+res = classifier(sent, candidate_labels)
+print(res)
+``` 
+### Acknowledgement
+The research work was supported by the Hellenic Foundation for Research and Innovation (HFRI) under the HFRI PhD Fellowship grant (Fellowship Number:50, 2nd call)
+
+### Citation info
+Citation for the Greek model TBA.
+Based on the work [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084)
+Kudos to @nreimers (Nils Reimers) for his support on Github . 
+ 

config.json

@@ -0,0 +1,39 @@
+{
+  "_name_or_path": "/home/earendil/Desktop/ML_playground/sentence-transformers/examples/training/cross-encoder/output/training_allnli-2021-09-18_19-10-56",
+  "architectures": [
+    "XLMRobertaForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1",
+    "2": "LABEL_2"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1,
+    "LABEL_2": 2
+  },
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "xlm-roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "output_past": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.10.0",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 250002
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a43dbce83e22dd86cabe5dcda54068f4f846ac6f3a1f6b2bed1a03d1ac38e3a4
+size 1112274377

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfc8146abe2a0488e9e2a0c56de7952f7c11ab059eca145a0a727afce0db2865
+size 5069051

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "sep_token": "</s>", "pad_token": "<pad>", "cls_token": "<s>", "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": false}}

tokenizer_config.json

@@ -0,0 +1 @@
+{"bos_token": "<s>", "eos_token": "</s>", "sep_token": "</s>", "cls_token": "<s>", "unk_token": "<unk>", "pad_token": "<pad>", "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "model_max_length": 512, "special_tokens_map_file": null, "name_or_path": "/home/earendil/Desktop/ML_playground/sentence-transformers/examples/training/cross-encoder/output/training_allnli-2021-09-18_19-10-56", "tokenizer_class": "XLMRobertaTokenizer"}