Add final trained model
Browse files- README.md +174 -162
- model.safetensors +1 -1
README.md
CHANGED
|
@@ -4,90 +4,102 @@ tags:
|
|
| 4 |
- sentence-similarity
|
| 5 |
- feature-extraction
|
| 6 |
- generated_from_trainer
|
| 7 |
-
- dataset_size:
|
| 8 |
- loss:MultipleNegativesRankingLoss
|
| 9 |
base_model: BAAI/bge-base-en-v1.5
|
| 10 |
widget:
|
| 11 |
-
- source_sentence: I
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
MyMemory Translation Memory API.
|
| 16 |
sentences:
|
| 17 |
-
- "def
|
| 18 |
-
\n\tArguments:\n\t---------\n\t\"\"\""
|
| 19 |
-
- "def
|
| 20 |
-
\
|
| 21 |
-
\
|
| 22 |
-
- "def
|
| 23 |
-
|
| 24 |
-
\
|
| 25 |
-
- source_sentence: I'm
|
| 26 |
-
|
| 27 |
-
|
|
|
|
|
|
|
| 28 |
sentences:
|
| 29 |
-
- "def
|
| 30 |
-
\
|
| 31 |
-
\t
|
| 32 |
-
\
|
| 33 |
-
|
| 34 |
-
\
|
| 35 |
-
\
|
| 36 |
-
\
|
| 37 |
-
\
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
\
|
| 42 |
-
|
| 43 |
-
\n\
|
| 44 |
-
\
|
| 45 |
-
|
| 46 |
-
\
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
sentences:
|
| 51 |
-
- "def
|
| 52 |
-
\
|
| 53 |
-
\t---------\n\t-
|
| 54 |
-
\t\
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
\
|
| 59 |
-
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
|
|
|
|
|
|
| 64 |
sentences:
|
| 65 |
-
- "def
|
| 66 |
-
\
|
| 67 |
-
\
|
| 68 |
-
|
| 69 |
-
\
|
| 70 |
-
\
|
| 71 |
-
\t-
|
| 72 |
-
|
| 73 |
-
\
|
| 74 |
-
\
|
| 75 |
-
|
| 76 |
-
- "def bbc_good_food_api_categories_collections_ids:\n\t\"\"\"\n\tDescription:\n\
|
| 77 |
-
\tGet all categories collection with there names and namd id\n\n\tArguments:\n\
|
| 78 |
\t---------\n\t\"\"\""
|
| 79 |
-
- source_sentence: I'm a
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
sentences:
|
| 84 |
-
- "def
|
| 85 |
-
\n\tArguments:\n\t---------\n\t\
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
\
|
| 89 |
-
|
| 90 |
-
\
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 91 |
pipeline_tag: sentence-similarity
|
| 92 |
library_name: sentence-transformers
|
| 93 |
metrics:
|
|
@@ -120,58 +132,58 @@ model-index:
|
|
| 120 |
type: dev
|
| 121 |
metrics:
|
| 122 |
- type: cosine_accuracy@1
|
| 123 |
-
value: 0.
|
| 124 |
name: Cosine Accuracy@1
|
| 125 |
- type: cosine_accuracy@3
|
| 126 |
-
value: 0.
|
| 127 |
name: Cosine Accuracy@3
|
| 128 |
- type: cosine_accuracy@5
|
| 129 |
-
value: 0.
|
| 130 |
name: Cosine Accuracy@5
|
| 131 |
- type: cosine_accuracy@10
|
| 132 |
-
value: 0.
|
| 133 |
name: Cosine Accuracy@10
|
| 134 |
- type: cosine_precision@1
|
| 135 |
-
value: 0.
|
| 136 |
name: Cosine Precision@1
|
| 137 |
- type: cosine_precision@3
|
| 138 |
-
value: 0.
|
| 139 |
name: Cosine Precision@3
|
| 140 |
- type: cosine_precision@5
|
| 141 |
-
value: 0.
|
| 142 |
name: Cosine Precision@5
|
| 143 |
- type: cosine_precision@10
|
| 144 |
-
value: 0.
|
| 145 |
name: Cosine Precision@10
|
| 146 |
- type: cosine_recall@1
|
| 147 |
-
value: 0.
|
| 148 |
name: Cosine Recall@1
|
| 149 |
- type: cosine_recall@3
|
| 150 |
-
value: 0.
|
| 151 |
name: Cosine Recall@3
|
| 152 |
- type: cosine_recall@5
|
| 153 |
-
value: 0.
|
| 154 |
name: Cosine Recall@5
|
| 155 |
- type: cosine_recall@10
|
| 156 |
-
value: 0.
|
| 157 |
name: Cosine Recall@10
|
| 158 |
- type: cosine_ndcg@1
|
| 159 |
-
value: 0.
|
| 160 |
name: Cosine Ndcg@1
|
| 161 |
- type: cosine_ndcg@3
|
| 162 |
-
value: 0.
|
| 163 |
name: Cosine Ndcg@3
|
| 164 |
- type: cosine_ndcg@5
|
| 165 |
-
value: 0.
|
| 166 |
name: Cosine Ndcg@5
|
| 167 |
- type: cosine_ndcg@10
|
| 168 |
-
value: 0.
|
| 169 |
name: Cosine Ndcg@10
|
| 170 |
- type: cosine_mrr@10
|
| 171 |
-
value: 0.
|
| 172 |
name: Cosine Mrr@10
|
| 173 |
- type: cosine_map@100
|
| 174 |
-
value: 0.
|
| 175 |
name: Cosine Map@100
|
| 176 |
---
|
| 177 |
|
|
@@ -225,9 +237,9 @@ from sentence_transformers import SentenceTransformer
|
|
| 225 |
model = SentenceTransformer("LorMolf/mnrl-toolbench-bge-base-en-v1.5")
|
| 226 |
# Run inference
|
| 227 |
sentences = [
|
| 228 |
-
"I'm a
|
| 229 |
-
'def
|
| 230 |
-
'def
|
| 231 |
]
|
| 232 |
embeddings = model.encode(sentences)
|
| 233 |
print(embeddings.shape)
|
|
@@ -272,26 +284,26 @@ You can finetune this model on your own dataset.
|
|
| 272 |
* Dataset: `dev`
|
| 273 |
* Evaluated with <code>src.port.retrieval_evaluator.DeviceAwareInformationRetrievalEvaluator</code>
|
| 274 |
|
| 275 |
-
| Metric | Value
|
| 276 |
-
|
| 277 |
-
| cosine_accuracy@1 | 0.
|
| 278 |
-
| cosine_accuracy@3 | 0.
|
| 279 |
-
| cosine_accuracy@5 | 0.
|
| 280 |
-
| cosine_accuracy@10 | 0.
|
| 281 |
-
| cosine_precision@1 | 0.
|
| 282 |
-
| cosine_precision@3 | 0.
|
| 283 |
-
| cosine_precision@5 | 0.
|
| 284 |
-
| cosine_precision@10 | 0.
|
| 285 |
-
| cosine_recall@1 | 0.
|
| 286 |
-
| cosine_recall@3 | 0.
|
| 287 |
-
| cosine_recall@5 | 0.
|
| 288 |
-
| cosine_recall@10 | 0.
|
| 289 |
-
| cosine_ndcg@1 | 0.
|
| 290 |
-
| cosine_ndcg@3 | 0.
|
| 291 |
-
| cosine_ndcg@5 | 0.
|
| 292 |
-
| **cosine_ndcg@10** | **0.
|
| 293 |
-
| cosine_mrr@10 | 0.
|
| 294 |
-
| cosine_map@100 | 0.
|
| 295 |
|
| 296 |
<!--
|
| 297 |
## Bias, Risks and Limitations
|
|
@@ -311,19 +323,19 @@ You can finetune this model on your own dataset.
|
|
| 311 |
|
| 312 |
#### Unnamed Dataset
|
| 313 |
|
| 314 |
-
* Size:
|
| 315 |
* Columns: <code>sentence_0</code>, <code>sentence_1</code>, and <code>sentence_2</code>
|
| 316 |
* Approximate statistics based on the first 1000 samples:
|
| 317 |
| | sentence_0 | sentence_1 | sentence_2 |
|
| 318 |
|:--------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
|
| 319 |
| type | string | string | string |
|
| 320 |
-
| details | <ul><li>min:
|
| 321 |
* Samples:
|
| 322 |
-
| sentence_0
|
| 323 |
-
|
| 324 |
-
| <code>I
|
| 325 |
-
| <code>I
|
| 326 |
-
| <code>I want to surprise my
|
| 327 |
* Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters:
|
| 328 |
```json
|
| 329 |
{
|
|
@@ -336,9 +348,9 @@ You can finetune this model on your own dataset.
|
|
| 336 |
#### Non-Default Hyperparameters
|
| 337 |
|
| 338 |
- `eval_strategy`: steps
|
| 339 |
-
- `per_device_train_batch_size`:
|
| 340 |
-
- `per_device_eval_batch_size`:
|
| 341 |
-
- `num_train_epochs`:
|
| 342 |
- `fp16`: True
|
| 343 |
- `multi_dataset_batch_sampler`: round_robin
|
| 344 |
|
|
@@ -349,8 +361,8 @@ You can finetune this model on your own dataset.
|
|
| 349 |
- `do_predict`: False
|
| 350 |
- `eval_strategy`: steps
|
| 351 |
- `prediction_loss_only`: True
|
| 352 |
-
- `per_device_train_batch_size`:
|
| 353 |
-
- `per_device_eval_batch_size`:
|
| 354 |
- `per_gpu_train_batch_size`: None
|
| 355 |
- `per_gpu_eval_batch_size`: None
|
| 356 |
- `gradient_accumulation_steps`: 1
|
|
@@ -362,7 +374,7 @@ You can finetune this model on your own dataset.
|
|
| 362 |
- `adam_beta2`: 0.999
|
| 363 |
- `adam_epsilon`: 1e-08
|
| 364 |
- `max_grad_norm`: 1.0
|
| 365 |
-
- `num_train_epochs`:
|
| 366 |
- `max_steps`: -1
|
| 367 |
- `lr_scheduler_type`: linear
|
| 368 |
- `lr_scheduler_kwargs`: {}
|
|
@@ -463,39 +475,39 @@ You can finetune this model on your own dataset.
|
|
| 463 |
</details>
|
| 464 |
|
| 465 |
### Training Logs
|
| 466 |
-
| Epoch | Step | dev_cosine_ndcg@10 |
|
| 467 |
-
|
| 468 |
-
| -1 | -1
|
| 469 |
-
| 0.
|
| 470 |
-
| 0.
|
| 471 |
-
| 0.
|
| 472 |
-
| 0.
|
| 473 |
-
| 0.
|
| 474 |
-
|
|
| 475 |
-
|
|
| 476 |
-
|
|
| 477 |
-
|
|
| 478 |
-
|
|
| 479 |
-
|
|
| 480 |
-
|
|
| 481 |
-
|
|
| 482 |
-
|
|
| 483 |
-
|
|
| 484 |
-
|
|
| 485 |
-
|
|
| 486 |
-
|
|
| 487 |
-
|
|
| 488 |
-
|
|
| 489 |
-
|
|
| 490 |
-
|
|
| 491 |
-
|
|
| 492 |
-
|
|
| 493 |
-
|
|
| 494 |
-
|
|
| 495 |
-
|
|
| 496 |
-
|
|
| 497 |
-
|
|
| 498 |
-
|
|
| 499 |
|
| 500 |
|
| 501 |
### Framework Versions
|
|
|
|
| 4 |
- sentence-similarity
|
| 5 |
- feature-extraction
|
| 6 |
- generated_from_trainer
|
| 7 |
+
- dataset_size:30000
|
| 8 |
- loss:MultipleNegativesRankingLoss
|
| 9 |
base_model: BAAI/bge-base-en-v1.5
|
| 10 |
widget:
|
| 11 |
+
- source_sentence: I need to plan a vacation from June 1st to June 10th, 2023 in the
|
| 12 |
+
United States. Can you provide me with a list of non-working days during this
|
| 13 |
+
period? Additionally, analyze the availability of events and obtain the details
|
| 14 |
+
of the first event on June 3rd. Also, check the responses for this event.
|
|
|
|
| 15 |
sentences:
|
| 16 |
+
- "def diablo4_smartable_getsponsorships:\n\t\"\"\"\n\tDescription:\n\tGet Diablo\
|
| 17 |
+
\ 4 sponsorships.\n\n\tArguments:\n\t---------\n\t\"\"\""
|
| 18 |
+
- "def YTStream_-_Download_YouTube_Videos.Download/Stream:\n\t\"\"\"\n\tDescription:\n\
|
| 19 |
+
\tStream or download info.\n\n\tArguments:\n\t---------\n\t- id : STRING (required)\n\
|
| 20 |
+
\t Description: Youtube Video Id.\n\t Default: UxxajLWwzqY\n\t\"\"\""
|
| 21 |
+
- "def 31Events_-_Send_Native_Calendar_Invites.EventGet:\n\t\"\"\"\n\t\n\n\tArguments:\n\
|
| 22 |
+
\t---------\n\t- event_id : STRING (required)\n\t Description: Event ID\n\t \
|
| 23 |
+
\ Default: 1\n\t\"\"\""
|
| 24 |
+
- source_sentence: I'm a student working on a research project about climate change.
|
| 25 |
+
Help me find some scientific articles and discussions on Reddit related to climate
|
| 26 |
+
change. Provide me with the top comments so that I can understand different perspectives.
|
| 27 |
+
Additionally, suggest some popular posts and their details that I can reference
|
| 28 |
+
in my project.
|
| 29 |
sentences:
|
| 30 |
+
- "def reddit_posts_by_username:\n\t\"\"\"\n\tDescription:\n\tPosts By Username\n\
|
| 31 |
+
\n\tArguments:\n\t---------\n\t- username : STRING (required)\n\t Default: GoldenChrysus\n\
|
| 32 |
+
\t- sort : STRING (required)\n\t Description: you can just send `new `or `hot`\n\
|
| 33 |
+
\t Default: new\n\t\"\"\""
|
| 34 |
+
- "def microsoft_translator_text_languages:\n\t\"\"\"\n\tDescription:\n\tGets the\
|
| 35 |
+
\ set of languages currently supported by other operations of the Translator Text\
|
| 36 |
+
\ API.\n\n\tArguments:\n\t---------\n\t- api-version : STRING (required)\n\t \
|
| 37 |
+
\ Description: Version of the API requested by the client. Value must be **3.0**.\n\
|
| 38 |
+
\t Default: 3.0\n\t\"\"\""
|
| 39 |
+
- "def socialgrep_post_search:\n\t\"\"\"\n\tDescription:\n\tSearches Reddit posts.\n\
|
| 40 |
+
\n\tArguments:\n\t---------\n\t- query : STRING (required)\n\t Description: The\
|
| 41 |
+
\ comma-separated query for the search. Supports the following term types:\n\t\
|
| 42 |
+
\n\t`site:{site_name}` - search only posts where the domain matches {site_name}.\n\
|
| 43 |
+
\t\n\t`-site:{site_name}` - search only posts where the domain does not match\
|
| 44 |
+
\ {site_name}.\n\t\n\t`/r/{subreddit}` - search only posts from the subreddit\
|
| 45 |
+
\ {subreddit}.\n\t\n\t`-/r/{subreddit}` - search only posts not from the subreddit\
|
| 46 |
+
\ {subreddit}.\n\t\n\t`{term}` - search only posts with titles containing the\
|
| 47 |
+
\ term {term}.\n\t\n\t`-{term}` - search only posts with titles not containing\
|
| 48 |
+
\ the term {term}.\n\t\n\t`score:{score}` - search only posts with score at least\
|
| 49 |
+
\ {score}.\n\t\n\t`before:{YYYY-mm-dd}`, `after:{YYYY-mm-dd}` - search only posts\
|
| 50 |
+
\ within the date range. `before` is inclusive, `after` is not.\n\t Default:\
|
| 51 |
+
\ /r/funny,cat\n\t\"\"\""
|
| 52 |
+
- source_sentence: I'm planning a weekend getaway with my friends and we want to watch
|
| 53 |
+
a football match. Can you provide me with the list of fixture IDs for the matches
|
| 54 |
+
scheduled for next month? Also, show me the league table and stats for the home
|
| 55 |
+
team of the match with ID 81930. Additionally, scrape the contacts from the website
|
| 56 |
+
of the home team to get their email and social media profiles.
|
| 57 |
sentences:
|
| 58 |
+
- "def coinranking_get_coin_exchanges:\n\t\"\"\"\n\tDescription:\n\tFind exchanges\
|
| 59 |
+
\ where a specific coin can be traded.\n\tThis endpoint requires the **ultra**\
|
| 60 |
+
\ plan or higher.\n\n\tArguments:\n\t---------\n\t- uuid : string (required)\n\
|
| 61 |
+
\t Description: UUID of the coin you want to request exchanges for\n\t Default:\
|
| 62 |
+
\ Qwsogvtv82FCd\n\t\"\"\""
|
| 63 |
+
- "def football_prediction_get_list_of_fixture_ids:\n\t\"\"\"\n\tDescription:\n\t\
|
| 64 |
+
Returns a list of fixture IDs that can be used to make requests to endpoints expecting\
|
| 65 |
+
\ a ID url parameter.\n\tCan be filtered by:\n\t\n\t- iso_date\n\t- market\n\t\
|
| 66 |
+
- federation\n\n\tArguments:\n\t---------\n\t\"\"\""
|
| 67 |
+
- "def open_brewery_db_breweries:\n\t\"\"\"\n\tDescription:\n\tList of Breweries\n\
|
| 68 |
+
\n\tArguments:\n\t---------\n\t\"\"\""
|
| 69 |
+
- source_sentence: I'm organizing a movie marathon and I need a mix of genres. Can
|
| 70 |
+
you recommend highly rated movies from various genres available on streaming services
|
| 71 |
+
like Netflix, Prime Video, Hulu, and Peacock? Additionally, provide me with the
|
| 72 |
+
TV schedule for tonight's movies.
|
| 73 |
sentences:
|
| 74 |
+
- "def streaming_availability_genres_free:\n\t\"\"\"\n\tDescription:\n\tGet the\
|
| 75 |
+
\ id to name mapping of supported genres.\n\n\tArguments:\n\t---------\n\t\"\"\
|
| 76 |
+
\""
|
| 77 |
+
- "def solcast_simple_pv_power_forecast:\n\t\"\"\"\n\tDescription:\n\tThe simple\
|
| 78 |
+
\ PV power request returns a first-guess PV power output forecast, based on your\
|
| 79 |
+
\ specified latitude and longitude plus some basic PV system characteristics.\n\
|
| 80 |
+
\n\tArguments:\n\t---------\n\t- capacity : NUMBER (required)\n\t Description:\
|
| 81 |
+
\ The capacity of the system, in Watts.\n\t Default: 0\n\t- latitude : NUMBER\
|
| 82 |
+
\ (required)\n\t Description: Latitude\n\t- longitude : NUMBER (required)\n\t\
|
| 83 |
+
\ Description: Longitude\n\t\"\"\""
|
| 84 |
+
- "def kargom_nerede_companies:\n\t\"\"\"\n\tDescription:\n\tCompanies\n\n\tArguments:\n\
|
|
|
|
|
|
|
| 85 |
\t---------\n\t\"\"\""
|
| 86 |
+
- source_sentence: I'm a food blogger and I need some interesting facts for my next
|
| 87 |
+
article. Fetch a random fact about a specific number and provide a historical
|
| 88 |
+
fact about a famous year. Additionally, recommend a genre of music to set the
|
| 89 |
+
mood for writing.
|
| 90 |
sentences:
|
| 91 |
+
- "def dicolink_get_lexical_field:\n\t\"\"\"\n\tDescription:\n\tGet Lexical Field\
|
| 92 |
+
\ for a word\n\n\tArguments:\n\t---------\n\t- mot : string (required)\n\t Default:\
|
| 93 |
+
\ cheval\n\t\"\"\""
|
| 94 |
+
- "def geodb_cities_places_near_location:\n\t\"\"\"\n\tDescription:\n\tGet places\
|
| 95 |
+
\ near the given location, filtering by optional criteria.\n\n\tArguments:\n\t\
|
| 96 |
+
---------\n\t- radius : STRING (required)\n\t Description: The location radius\
|
| 97 |
+
\ within which to find places\n\t Default: 100\n\t- locationid : STRING (required)\n\
|
| 98 |
+
\t Description: Only cities near this location. Latitude/longitude in ISO-6709\
|
| 99 |
+
\ format: ±DD.DDDD±DDD.DDDD\n\t Default: 33.832213-118.387099\n\t\"\"\""
|
| 100 |
+
- "def deezer_genre:\n\t\"\"\"\n\tDescription:\n\tA genre object\n\n\tArguments:\n\
|
| 101 |
+
\t---------\n\t- id : STRING (required)\n\t Description: The editorial's Deezer\
|
| 102 |
+
\ id\n\t\"\"\""
|
| 103 |
pipeline_tag: sentence-similarity
|
| 104 |
library_name: sentence-transformers
|
| 105 |
metrics:
|
|
|
|
| 132 |
type: dev
|
| 133 |
metrics:
|
| 134 |
- type: cosine_accuracy@1
|
| 135 |
+
value: 0.66
|
| 136 |
name: Cosine Accuracy@1
|
| 137 |
- type: cosine_accuracy@3
|
| 138 |
+
value: 0.82
|
| 139 |
name: Cosine Accuracy@3
|
| 140 |
- type: cosine_accuracy@5
|
| 141 |
+
value: 0.88
|
| 142 |
name: Cosine Accuracy@5
|
| 143 |
- type: cosine_accuracy@10
|
| 144 |
+
value: 0.95
|
| 145 |
name: Cosine Accuracy@10
|
| 146 |
- type: cosine_precision@1
|
| 147 |
+
value: 0.66
|
| 148 |
name: Cosine Precision@1
|
| 149 |
- type: cosine_precision@3
|
| 150 |
+
value: 0.4833333333333333
|
| 151 |
name: Cosine Precision@3
|
| 152 |
- type: cosine_precision@5
|
| 153 |
+
value: 0.3600000000000001
|
| 154 |
name: Cosine Precision@5
|
| 155 |
- type: cosine_precision@10
|
| 156 |
+
value: 0.21800000000000005
|
| 157 |
name: Cosine Precision@10
|
| 158 |
- type: cosine_recall@1
|
| 159 |
+
value: 0.25066666666666665
|
| 160 |
name: Cosine Recall@1
|
| 161 |
- type: cosine_recall@3
|
| 162 |
+
value: 0.5509999999999999
|
| 163 |
name: Cosine Recall@3
|
| 164 |
- type: cosine_recall@5
|
| 165 |
+
value: 0.6613333333333332
|
| 166 |
name: Cosine Recall@5
|
| 167 |
- type: cosine_recall@10
|
| 168 |
+
value: 0.7801666666666667
|
| 169 |
name: Cosine Recall@10
|
| 170 |
- type: cosine_ndcg@1
|
| 171 |
+
value: 0.66
|
| 172 |
name: Cosine Ndcg@1
|
| 173 |
- type: cosine_ndcg@3
|
| 174 |
+
value: 0.582592770063282
|
| 175 |
name: Cosine Ndcg@3
|
| 176 |
- type: cosine_ndcg@5
|
| 177 |
+
value: 0.633788337516139
|
| 178 |
name: Cosine Ndcg@5
|
| 179 |
- type: cosine_ndcg@10
|
| 180 |
+
value: 0.6889055410848939
|
| 181 |
name: Cosine Ndcg@10
|
| 182 |
- type: cosine_mrr@10
|
| 183 |
+
value: 0.7467063492063493
|
| 184 |
name: Cosine Mrr@10
|
| 185 |
- type: cosine_map@100
|
| 186 |
+
value: 0.629168458376448
|
| 187 |
name: Cosine Map@100
|
| 188 |
---
|
| 189 |
|
|
|
|
| 237 |
model = SentenceTransformer("LorMolf/mnrl-toolbench-bge-base-en-v1.5")
|
| 238 |
# Run inference
|
| 239 |
sentences = [
|
| 240 |
+
"I'm a food blogger and I need some interesting facts for my next article. Fetch a random fact about a specific number and provide a historical fact about a famous year. Additionally, recommend a genre of music to set the mood for writing.",
|
| 241 |
+
'def deezer_genre:\n\t"""\n\tDescription:\n\tA genre object\n\n\tArguments:\n\t---------\n\t- id : STRING (required)\n\t Description: The editorial\'s Deezer id\n\t"""',
|
| 242 |
+
'def dicolink_get_lexical_field:\n\t"""\n\tDescription:\n\tGet Lexical Field for a word\n\n\tArguments:\n\t---------\n\t- mot : string (required)\n\t Default: cheval\n\t"""',
|
| 243 |
]
|
| 244 |
embeddings = model.encode(sentences)
|
| 245 |
print(embeddings.shape)
|
|
|
|
| 284 |
* Dataset: `dev`
|
| 285 |
* Evaluated with <code>src.port.retrieval_evaluator.DeviceAwareInformationRetrievalEvaluator</code>
|
| 286 |
|
| 287 |
+
| Metric | Value |
|
| 288 |
+
|:--------------------|:-----------|
|
| 289 |
+
| cosine_accuracy@1 | 0.66 |
|
| 290 |
+
| cosine_accuracy@3 | 0.82 |
|
| 291 |
+
| cosine_accuracy@5 | 0.88 |
|
| 292 |
+
| cosine_accuracy@10 | 0.95 |
|
| 293 |
+
| cosine_precision@1 | 0.66 |
|
| 294 |
+
| cosine_precision@3 | 0.4833 |
|
| 295 |
+
| cosine_precision@5 | 0.36 |
|
| 296 |
+
| cosine_precision@10 | 0.218 |
|
| 297 |
+
| cosine_recall@1 | 0.2507 |
|
| 298 |
+
| cosine_recall@3 | 0.551 |
|
| 299 |
+
| cosine_recall@5 | 0.6613 |
|
| 300 |
+
| cosine_recall@10 | 0.7802 |
|
| 301 |
+
| cosine_ndcg@1 | 0.66 |
|
| 302 |
+
| cosine_ndcg@3 | 0.5826 |
|
| 303 |
+
| cosine_ndcg@5 | 0.6338 |
|
| 304 |
+
| **cosine_ndcg@10** | **0.6889** |
|
| 305 |
+
| cosine_mrr@10 | 0.7467 |
|
| 306 |
+
| cosine_map@100 | 0.6292 |
|
| 307 |
|
| 308 |
<!--
|
| 309 |
## Bias, Risks and Limitations
|
|
|
|
| 323 |
|
| 324 |
#### Unnamed Dataset
|
| 325 |
|
| 326 |
+
* Size: 30,000 training samples
|
| 327 |
* Columns: <code>sentence_0</code>, <code>sentence_1</code>, and <code>sentence_2</code>
|
| 328 |
* Approximate statistics based on the first 1000 samples:
|
| 329 |
| | sentence_0 | sentence_1 | sentence_2 |
|
| 330 |
|:--------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
|
| 331 |
| type | string | string | string |
|
| 332 |
+
| details | <ul><li>min: 29 tokens</li><li>mean: 61.36 tokens</li><li>max: 133 tokens</li></ul> | <ul><li>min: 28 tokens</li><li>mean: 82.89 tokens</li><li>max: 512 tokens</li></ul> | <ul><li>min: 26 tokens</li><li>mean: 85.81 tokens</li><li>max: 512 tokens</li></ul> |
|
| 333 |
* Samples:
|
| 334 |
+
| sentence_0 | sentence_1 | sentence_2 |
|
| 335 |
+
|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
| 336 |
+
| <code>My family and I are planning a beach vacation in Florida next month. Can you provide us with the current weather conditions, active alerts, and station details for Miami and Orlando? Additionally, we would like a radiation forecast to plan our outdoor activities.</code> | <code>def solcast_simple_radiation_forecast:<br> """<br> Description:<br> The simple radiation request returns detailed solar radiation data for the next week based only on your latitude and longitude.<br><br> Arguments:<br> ---------<br> - latitude : NUMBER (required)<br> Description: Latitude<br> - longitude : NUMBER (required)<br> Description: Longitude<br> """</code> | <code>def uk_boundaries_io_retrieve_uk_postal_district_outline_boundaries:<br> """<br> Description:<br> example: Query by "TW12" district<br><br> Arguments:<br> ---------<br> - postal-district : STRING (required)<br> Description: Query by postal district code.<br> Default: TW12<br> """</code> |
|
| 337 |
+
| <code>I am planning a vacation to a tropical destination and I need some information to make the most of my trip. Can you please provide me with the current weather data for a location with latitude 25.5 and longitude -80.5? Additionally, I would like a 16-day forecast for this location. Furthermore, I am interested in knowing the available cities in the country associated with this location. Lastly, please suggest some popular tourist attractions in this country.</code> | <code>def weather_forecast_14_days_list_of_cities_in_one_country:<br> """<br> Description:<br> List of cities in one Country<br><br> Arguments:<br> ---------<br> """</code> | <code>def Billboard-API.Brazil_Songs:<br> """<br> Description:<br> Provide the Brazil Songs chart information<br><br> Arguments:<br> ---------<br> - date : DATE (YYYY-MM-DD) (required)<br> Description: date format(YYYY-MM-DD)<br> Default: 2022-05-07<br> - range : STRING (required)<br> Default: 1-10<br> """</code> |
|
| 338 |
+
| <code>I want to surprise my family with a special dinner tonight. Can you suggest some quick and easy recipes for a main course? Also, provide me with the list of ingredients required for each recipe. Additionally, I would like to know the plant hardiness zone for our area, which is zip code 90210.</code> | <code>def yummly_feeds_list:<br> """<br> Description:<br> List feeds by category<br><br> Arguments:<br> ---------<br> - start : NUMBER (required)<br> Description: The offset of items to be ignored in response for paging<br> Default: 0<br> - limit : NUMBER (required)<br> Description: Number of items returned per response<br> Default: 24<br> """</code> | <code>def line_messaging_get_number_of_sent_reply_messages:<br> """<br> Description:<br> Gets the number of messages sent with the /bot/message/reply endpoint.<br><br> Arguments:<br> ---------<br> - date : STRING (required)<br> Description: Date the messages were sent. Format: yyyyMMdd (Example: 20191231) Timezone: UTC+9<br> """</code> |
|
| 339 |
* Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters:
|
| 340 |
```json
|
| 341 |
{
|
|
|
|
| 348 |
#### Non-Default Hyperparameters
|
| 349 |
|
| 350 |
- `eval_strategy`: steps
|
| 351 |
+
- `per_device_train_batch_size`: 2
|
| 352 |
+
- `per_device_eval_batch_size`: 2
|
| 353 |
+
- `num_train_epochs`: 1
|
| 354 |
- `fp16`: True
|
| 355 |
- `multi_dataset_batch_sampler`: round_robin
|
| 356 |
|
|
|
|
| 361 |
- `do_predict`: False
|
| 362 |
- `eval_strategy`: steps
|
| 363 |
- `prediction_loss_only`: True
|
| 364 |
+
- `per_device_train_batch_size`: 2
|
| 365 |
+
- `per_device_eval_batch_size`: 2
|
| 366 |
- `per_gpu_train_batch_size`: None
|
| 367 |
- `per_gpu_eval_batch_size`: None
|
| 368 |
- `gradient_accumulation_steps`: 1
|
|
|
|
| 374 |
- `adam_beta2`: 0.999
|
| 375 |
- `adam_epsilon`: 1e-08
|
| 376 |
- `max_grad_norm`: 1.0
|
| 377 |
+
- `num_train_epochs`: 1
|
| 378 |
- `max_steps`: -1
|
| 379 |
- `lr_scheduler_type`: linear
|
| 380 |
- `lr_scheduler_kwargs`: {}
|
|
|
|
| 475 |
</details>
|
| 476 |
|
| 477 |
### Training Logs
|
| 478 |
+
| Epoch | Step | Training Loss | dev_cosine_ndcg@10 |
|
| 479 |
+
|:------:|:-----:|:-------------:|:------------------:|
|
| 480 |
+
| -1 | -1 | - | 0.6031 |
|
| 481 |
+
| 0.0333 | 500 | 0.3188 | - |
|
| 482 |
+
| 0.0667 | 1000 | 0.2079 | - |
|
| 483 |
+
| 0.1 | 1500 | 0.2178 | - |
|
| 484 |
+
| 0.1333 | 2000 | 0.186 | - |
|
| 485 |
+
| 0.1667 | 2500 | 0.1665 | - |
|
| 486 |
+
| 0.2 | 3000 | 0.205 | 0.6953 |
|
| 487 |
+
| 0.2333 | 3500 | 0.149 | - |
|
| 488 |
+
| 0.2667 | 4000 | 0.1691 | - |
|
| 489 |
+
| 0.3 | 4500 | 0.1703 | - |
|
| 490 |
+
| 0.3333 | 5000 | 0.1588 | - |
|
| 491 |
+
| 0.3667 | 5500 | 0.1348 | - |
|
| 492 |
+
| 0.4 | 6000 | 0.1625 | 0.6639 |
|
| 493 |
+
| 0.4333 | 6500 | 0.1415 | - |
|
| 494 |
+
| 0.4667 | 7000 | 0.13 | - |
|
| 495 |
+
| 0.5 | 7500 | 0.1271 | - |
|
| 496 |
+
| 0.5333 | 8000 | 0.1058 | - |
|
| 497 |
+
| 0.5667 | 8500 | 0.1031 | - |
|
| 498 |
+
| 0.6 | 9000 | 0.1026 | 0.6860 |
|
| 499 |
+
| 0.6333 | 9500 | 0.1031 | - |
|
| 500 |
+
| 0.6667 | 10000 | 0.1248 | - |
|
| 501 |
+
| 0.7 | 10500 | 0.0909 | - |
|
| 502 |
+
| 0.7333 | 11000 | 0.1055 | - |
|
| 503 |
+
| 0.7667 | 11500 | 0.101 | - |
|
| 504 |
+
| 0.8 | 12000 | 0.0598 | 0.6778 |
|
| 505 |
+
| 0.8333 | 12500 | 0.0949 | - |
|
| 506 |
+
| 0.8667 | 13000 | 0.062 | - |
|
| 507 |
+
| 0.9 | 13500 | 0.1129 | - |
|
| 508 |
+
| 0.9333 | 14000 | 0.1106 | - |
|
| 509 |
+
| 0.9667 | 14500 | 0.0653 | - |
|
| 510 |
+
| 1.0 | 15000 | 0.0669 | 0.6889 |
|
| 511 |
|
| 512 |
|
| 513 |
### Framework Versions
|
model.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 437951328
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0954f24844b3b1cbe528871e2f850444bb5055667fb0ecf42b44a639a482209c
|
| 3 |
size 437951328
|