Add new SentenceTransformer model
Browse files- README.md +137 -144
- model.safetensors +1 -1
README.md
CHANGED
@@ -12,82 +12,75 @@ tags:
|
|
12 |
- loss:MultipleNegativesRankingLoss
|
13 |
base_model: nomic-ai/modernbert-embed-base
|
14 |
widget:
|
15 |
-
- source_sentence: What
|
16 |
-
mentioned?
|
17 |
sentences:
|
18 |
-
- '1. Chatbots and Virtual Assistants
|
19 |
-
|
20 |
-
One of the most visible LLM integrations is in chatbots. Tools like ChatGPT, Claude,
|
21 |
-
and Bard are themselves chatbot interfaces built on LLMs. Many businesses are
|
22 |
-
now integrating these models into their websites and customer support systems.'
|
23 |
-
- For example, e-commerce websites can deploy LLM-powered assistants to help customers
|
24 |
-
find products, track orders, or get personalized recommendations—much more effectively
|
25 |
-
than traditional rule-based bots.
|
26 |
-
- Some services, like ColBERT, Marqo, and ColQwen, specialize in integrating LLMs
|
27 |
-
into search pipelines for both text and multi-modal (text + image) content.
|
28 |
-
- source_sentence: What is one method mentioned for deploying LLMs?
|
29 |
-
sentences:
|
30 |
-
- However, deploying LLMs effectively in real-world applications often requires
|
31 |
-
LLM integration. This means embedding these models into systems, workflows, or
|
32 |
-
products where they can interact with other components like databases, APIs, user
|
33 |
-
interfaces, or even custom business logic
|
34 |
-
- Some services, like ColBERT, Marqo, and ColQwen, specialize in integrating LLMs
|
35 |
-
into search pipelines for both text and multi-modal (text + image) content.
|
36 |
-
- However, deploying LLMs effectively in real-world applications often requires
|
37 |
-
LLM integration. This means embedding these models into systems, workflows, or
|
38 |
-
products where they can interact with other components like databases, APIs, user
|
39 |
-
interfaces, or even custom business logic
|
40 |
-
- source_sentence: What will an LLM likely respond with when prompted about the capital
|
41 |
-
of France?
|
42 |
-
sentences:
|
43 |
-
- . For instance, a spam filter doesn’t just block emails with specific keywords—it
|
44 |
-
learns from thousands of examples what spam typically looks like.
|
45 |
- Over the past few years, the field of ML has advanced rapidly, especially in the
|
46 |
area of Natural Language Processing (NLP)—the ability of machines to understand
|
47 |
and generate human language. At the forefront of this progress are Large Language
|
48 |
Models (LLMs), such as OpenAI’s GPT (Generative Pre-trained Transformer), Google’s
|
49 |
PaLM, and Meta’s LLaMA
|
50 |
-
- For example, given a prompt like "The capital of France is", an LLM trained on
|
51 |
-
a wide range of texts will likely respond with "Paris". But beyond trivia, LLMs
|
52 |
-
can write essays, draft emails, simulate conversations, generate code snippets,
|
53 |
-
and much more.
|
54 |
-
- source_sentence: What might an LLM be connected to in a customer support chatbot?
|
55 |
-
sentences:
|
56 |
-
- . For instance, a spam filter doesn’t just block emails with specific keywords—it
|
57 |
-
learns from thousands of examples what spam typically looks like.
|
58 |
- . For example, integrating an LLM into a customer support chatbot might involve
|
59 |
connecting it to a company’s internal knowledge base, enabling it to answer customer
|
60 |
questions using accurate, up-to-date information.
|
61 |
-
-
|
62 |
-
|
|
|
|
|
|
|
|
|
63 |
sentences:
|
64 |
-
-
|
65 |
-
|
66 |
-
|
67 |
-
|
68 |
-
|
69 |
-
compliance concerns.
|
70 |
-
|
71 |
-
|
72 |
-
Cost and Latency: Running LLMs, especially large ones, can be computationally
|
73 |
-
expensive and slow.'
|
74 |
-
- '6. APIs and Developer Tools
|
75 |
-
|
76 |
-
Developers can integrate LLMs into their own apps using APIs provided by companies
|
77 |
-
like OpenAI, Anthropic, and Cohere. These APIs allow developers to send prompts
|
78 |
-
and receive intelligent outputs in return.
|
79 |
|
|
|
|
|
|
|
80 |
|
81 |
-
This enables custom applications like:
|
82 |
|
|
|
83 |
|
84 |
-
|
|
|
|
|
|
|
|
|
|
|
85 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
86 |
|
87 |
-
|
|
|
|
|
88 |
|
89 |
|
90 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
91 |
- '5. Education and Learning Platforms
|
92 |
|
93 |
Educational tools like Khanmigo (from Khan Academy) and other tutoring platforms
|
@@ -122,49 +115,49 @@ model-index:
|
|
122 |
type: dim_768
|
123 |
metrics:
|
124 |
- type: cosine_accuracy@1
|
125 |
-
value: 0.
|
126 |
name: Cosine Accuracy@1
|
127 |
- type: cosine_accuracy@3
|
128 |
-
value: 0.
|
129 |
name: Cosine Accuracy@3
|
130 |
- type: cosine_accuracy@5
|
131 |
-
value: 0
|
132 |
name: Cosine Accuracy@5
|
133 |
- type: cosine_accuracy@10
|
134 |
value: 1.0
|
135 |
name: Cosine Accuracy@10
|
136 |
- type: cosine_precision@1
|
137 |
-
value: 0.
|
138 |
name: Cosine Precision@1
|
139 |
- type: cosine_precision@3
|
140 |
-
value: 0.
|
141 |
name: Cosine Precision@3
|
142 |
- type: cosine_precision@5
|
143 |
-
value: 0.
|
144 |
name: Cosine Precision@5
|
145 |
- type: cosine_precision@10
|
146 |
value: 0.10000000000000003
|
147 |
name: Cosine Precision@10
|
148 |
- type: cosine_recall@1
|
149 |
-
value: 0.
|
150 |
name: Cosine Recall@1
|
151 |
- type: cosine_recall@3
|
152 |
-
value: 0.
|
153 |
name: Cosine Recall@3
|
154 |
- type: cosine_recall@5
|
155 |
-
value: 0
|
156 |
name: Cosine Recall@5
|
157 |
- type: cosine_recall@10
|
158 |
value: 1.0
|
159 |
name: Cosine Recall@10
|
160 |
- type: cosine_ndcg@10
|
161 |
-
value: 0.
|
162 |
name: Cosine Ndcg@10
|
163 |
- type: cosine_mrr@10
|
164 |
-
value: 0.
|
165 |
name: Cosine Mrr@10
|
166 |
- type: cosine_map@100
|
167 |
-
value: 0.
|
168 |
name: Cosine Map@100
|
169 |
- task:
|
170 |
type: information-retrieval
|
@@ -180,10 +173,10 @@ model-index:
|
|
180 |
value: 0.8
|
181 |
name: Cosine Accuracy@3
|
182 |
- type: cosine_accuracy@5
|
183 |
-
value: 0.
|
184 |
name: Cosine Accuracy@5
|
185 |
- type: cosine_accuracy@10
|
186 |
-
value: 0
|
187 |
name: Cosine Accuracy@10
|
188 |
- type: cosine_precision@1
|
189 |
value: 0.6666666666666666
|
@@ -192,10 +185,10 @@ model-index:
|
|
192 |
value: 0.2666666666666667
|
193 |
name: Cosine Precision@3
|
194 |
- type: cosine_precision@5
|
195 |
-
value: 0.
|
196 |
name: Cosine Precision@5
|
197 |
- type: cosine_precision@10
|
198 |
-
value: 0.
|
199 |
name: Cosine Precision@10
|
200 |
- type: cosine_recall@1
|
201 |
value: 0.6666666666666666
|
@@ -204,19 +197,19 @@ model-index:
|
|
204 |
value: 0.8
|
205 |
name: Cosine Recall@3
|
206 |
- type: cosine_recall@5
|
207 |
-
value: 0.
|
208 |
name: Cosine Recall@5
|
209 |
- type: cosine_recall@10
|
210 |
-
value: 0
|
211 |
name: Cosine Recall@10
|
212 |
- type: cosine_ndcg@10
|
213 |
-
value: 0.
|
214 |
name: Cosine Ndcg@10
|
215 |
- type: cosine_mrr@10
|
216 |
-
value: 0.
|
217 |
name: Cosine Mrr@10
|
218 |
- type: cosine_map@100
|
219 |
-
value: 0.
|
220 |
name: Cosine Map@100
|
221 |
- task:
|
222 |
type: information-retrieval
|
@@ -229,10 +222,10 @@ model-index:
|
|
229 |
value: 0.6666666666666666
|
230 |
name: Cosine Accuracy@1
|
231 |
- type: cosine_accuracy@3
|
232 |
-
value: 0.
|
233 |
name: Cosine Accuracy@3
|
234 |
- type: cosine_accuracy@5
|
235 |
-
value: 0
|
236 |
name: Cosine Accuracy@5
|
237 |
- type: cosine_accuracy@10
|
238 |
value: 1.0
|
@@ -241,10 +234,10 @@ model-index:
|
|
241 |
value: 0.6666666666666666
|
242 |
name: Cosine Precision@1
|
243 |
- type: cosine_precision@3
|
244 |
-
value: 0.
|
245 |
name: Cosine Precision@3
|
246 |
- type: cosine_precision@5
|
247 |
-
value: 0.
|
248 |
name: Cosine Precision@5
|
249 |
- type: cosine_precision@10
|
250 |
value: 0.10000000000000003
|
@@ -253,22 +246,22 @@ model-index:
|
|
253 |
value: 0.6666666666666666
|
254 |
name: Cosine Recall@1
|
255 |
- type: cosine_recall@3
|
256 |
-
value: 0.
|
257 |
name: Cosine Recall@3
|
258 |
- type: cosine_recall@5
|
259 |
-
value: 0
|
260 |
name: Cosine Recall@5
|
261 |
- type: cosine_recall@10
|
262 |
value: 1.0
|
263 |
name: Cosine Recall@10
|
264 |
- type: cosine_ndcg@10
|
265 |
-
value: 0.
|
266 |
name: Cosine Ndcg@10
|
267 |
- type: cosine_mrr@10
|
268 |
-
value: 0.
|
269 |
name: Cosine Mrr@10
|
270 |
- type: cosine_map@100
|
271 |
-
value: 0.
|
272 |
name: Cosine Map@100
|
273 |
- task:
|
274 |
type: information-retrieval
|
@@ -278,49 +271,49 @@ model-index:
|
|
278 |
type: dim_128
|
279 |
metrics:
|
280 |
- type: cosine_accuracy@1
|
281 |
-
value: 0.
|
282 |
name: Cosine Accuracy@1
|
283 |
- type: cosine_accuracy@3
|
284 |
-
value: 0.
|
285 |
name: Cosine Accuracy@3
|
286 |
- type: cosine_accuracy@5
|
287 |
value: 0.8
|
288 |
name: Cosine Accuracy@5
|
289 |
- type: cosine_accuracy@10
|
290 |
-
value: 0.
|
291 |
name: Cosine Accuracy@10
|
292 |
- type: cosine_precision@1
|
293 |
-
value: 0.
|
294 |
name: Cosine Precision@1
|
295 |
- type: cosine_precision@3
|
296 |
-
value: 0.
|
297 |
name: Cosine Precision@3
|
298 |
- type: cosine_precision@5
|
299 |
value: 0.16000000000000003
|
300 |
name: Cosine Precision@5
|
301 |
- type: cosine_precision@10
|
302 |
-
value: 0.
|
303 |
name: Cosine Precision@10
|
304 |
- type: cosine_recall@1
|
305 |
-
value: 0.
|
306 |
name: Cosine Recall@1
|
307 |
- type: cosine_recall@3
|
308 |
-
value: 0.
|
309 |
name: Cosine Recall@3
|
310 |
- type: cosine_recall@5
|
311 |
value: 0.8
|
312 |
name: Cosine Recall@5
|
313 |
- type: cosine_recall@10
|
314 |
-
value: 0.
|
315 |
name: Cosine Recall@10
|
316 |
- type: cosine_ndcg@10
|
317 |
-
value: 0.
|
318 |
name: Cosine Ndcg@10
|
319 |
- type: cosine_mrr@10
|
320 |
-
value: 0.
|
321 |
name: Cosine Mrr@10
|
322 |
- type: cosine_map@100
|
323 |
-
value: 0.
|
324 |
name: Cosine Map@100
|
325 |
- task:
|
326 |
type: information-retrieval
|
@@ -330,49 +323,49 @@ model-index:
|
|
330 |
type: dim_64
|
331 |
metrics:
|
332 |
- type: cosine_accuracy@1
|
333 |
-
value: 0.
|
334 |
name: Cosine Accuracy@1
|
335 |
- type: cosine_accuracy@3
|
336 |
-
value: 0.
|
337 |
name: Cosine Accuracy@3
|
338 |
- type: cosine_accuracy@5
|
339 |
value: 0.8
|
340 |
name: Cosine Accuracy@5
|
341 |
- type: cosine_accuracy@10
|
342 |
-
value: 0.
|
343 |
name: Cosine Accuracy@10
|
344 |
- type: cosine_precision@1
|
345 |
-
value: 0.
|
346 |
name: Cosine Precision@1
|
347 |
- type: cosine_precision@3
|
348 |
-
value: 0.
|
349 |
name: Cosine Precision@3
|
350 |
- type: cosine_precision@5
|
351 |
value: 0.16000000000000003
|
352 |
name: Cosine Precision@5
|
353 |
- type: cosine_precision@10
|
354 |
-
value: 0.
|
355 |
name: Cosine Precision@10
|
356 |
- type: cosine_recall@1
|
357 |
-
value: 0.
|
358 |
name: Cosine Recall@1
|
359 |
- type: cosine_recall@3
|
360 |
-
value: 0.
|
361 |
name: Cosine Recall@3
|
362 |
- type: cosine_recall@5
|
363 |
value: 0.8
|
364 |
name: Cosine Recall@5
|
365 |
- type: cosine_recall@10
|
366 |
-
value: 0.
|
367 |
name: Cosine Recall@10
|
368 |
- type: cosine_ndcg@10
|
369 |
-
value: 0.
|
370 |
name: Cosine Ndcg@10
|
371 |
- type: cosine_mrr@10
|
372 |
-
value: 0.
|
373 |
name: Cosine Mrr@10
|
374 |
- type: cosine_map@100
|
375 |
-
value: 0.
|
376 |
name: Cosine Map@100
|
377 |
---
|
378 |
|
@@ -428,7 +421,7 @@ model = SentenceTransformer("Nuf-hugginface/modernbert-embed-quickb")
|
|
428 |
sentences = [
|
429 |
'What type of dialogues can LLMs simulate?',
|
430 |
'5. Education and Learning Platforms\nEducational tools like Khanmigo (from Khan Academy) and other tutoring platforms are leveraging LLMs to provide real-time help to students. LLMs can break down complex topics, provide feedback on writing, and simulate Socratic-style dialogues.',
|
431 |
-
'
|
432 |
]
|
433 |
embeddings = model.encode(sentences)
|
434 |
print(embeddings.shape)
|
@@ -475,21 +468,21 @@ You can finetune this model on your own dataset.
|
|
475 |
|
476 |
| Metric | dim_768 | dim_512 | dim_256 | dim_128 | dim_64 |
|
477 |
|:--------------------|:-----------|:-----------|:-----------|:-----------|:-----------|
|
478 |
-
| cosine_accuracy@1 | 0.
|
479 |
-
| cosine_accuracy@3 | 0.
|
480 |
-
| cosine_accuracy@5 | 0.8667 |
|
481 |
-
| cosine_accuracy@10 | 1.0 | 0
|
482 |
-
| cosine_precision@1 | 0.
|
483 |
-
| cosine_precision@3 | 0.
|
484 |
-
| cosine_precision@5 | 0.
|
485 |
-
| cosine_precision@10 | 0.1 | 0.
|
486 |
-
| cosine_recall@1 | 0.
|
487 |
-
| cosine_recall@3 | 0.
|
488 |
-
| cosine_recall@5 | 0.8667 |
|
489 |
-
| cosine_recall@10 | 1.0 | 0
|
490 |
-
| **cosine_ndcg@10** | **0.
|
491 |
-
| cosine_mrr@10 | 0.
|
492 |
-
| cosine_map@100 | 0.
|
493 |
|
494 |
<!--
|
495 |
## Bias, Risks and Limitations
|
@@ -512,16 +505,16 @@ You can finetune this model on your own dataset.
|
|
512 |
* Size: 127 training samples
|
513 |
* Columns: <code>anchor</code> and <code>positive</code>
|
514 |
* Approximate statistics based on the first 127 samples:
|
515 |
-
| | anchor
|
516 |
-
|
517 |
-
| type | string
|
518 |
-
| details | <ul><li>min: 8 tokens</li><li>mean: 13.
|
519 |
* Samples:
|
520 |
-
| anchor
|
521 |
-
|
522 |
-
| <code>What
|
523 |
-
| <code>What
|
524 |
-
| <code>
|
525 |
* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
|
526 |
```json
|
527 |
{
|
@@ -680,13 +673,13 @@ You can finetune this model on your own dataset.
|
|
680 |
</details>
|
681 |
|
682 |
### Training Logs
|
683 |
-
| Epoch | Step
|
684 |
-
|
685 |
-
| 1.0 | 4
|
686 |
-
|
|
687 |
-
| 2.5 | 10
|
688 |
-
| 3.0 | 12
|
689 |
-
| 4.0
|
690 |
|
691 |
* The bold row denotes the saved checkpoint.
|
692 |
|
|
|
12 |
- loss:MultipleNegativesRankingLoss
|
13 |
base_model: nomic-ai/modernbert-embed-base
|
14 |
widget:
|
15 |
+
- source_sentence: What is the difference between traditional programming and ML?
|
|
|
16 |
sentences:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
17 |
- Over the past few years, the field of ML has advanced rapidly, especially in the
|
18 |
area of Natural Language Processing (NLP)—the ability of machines to understand
|
19 |
and generate human language. At the forefront of this progress are Large Language
|
20 |
Models (LLMs), such as OpenAI’s GPT (Generative Pre-trained Transformer), Google’s
|
21 |
PaLM, and Meta’s LLaMA
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
22 |
- . For example, integrating an LLM into a customer support chatbot might involve
|
23 |
connecting it to a company’s internal knowledge base, enabling it to answer customer
|
24 |
questions using accurate, up-to-date information.
|
25 |
+
- A major subset of AI is Machine Learning (ML), which involves algorithms that
|
26 |
+
learn from data rather than being explicitly programmed. Instead of writing detailed
|
27 |
+
instructions for every task, ML models find patterns in large datasets and use
|
28 |
+
these patterns to make predictions or decisions
|
29 |
+
- source_sentence: What is one of the tasks mentioned that involves creating new written
|
30 |
+
content?
|
31 |
sentences:
|
32 |
+
- In summary, AI and ML form the foundation for intelligent automation, while LLMs
|
33 |
+
represent a breakthrough in language understanding and generation. Integrating
|
34 |
+
these models into real-world systems unlocks practical value, turning raw intelligence
|
35 |
+
into tangible solutions
|
36 |
+
- '8. Security and Compliance Integrations
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
37 |
|
38 |
+
Some organizations are integrating LLMs to detect anomalies in text communications
|
39 |
+
(e.g., phishing detection or policy violations). LLMs can analyze language usage
|
40 |
+
and flag potentially suspicious behavior more flexibly than keyword-based filters.
|
41 |
|
|
|
42 |
|
43 |
+
Challenges in LLM Integration
|
44 |
|
45 |
+
Despite their promise, integrating LLMs comes with challenges:'
|
46 |
+
- . These include text generation, summarization, translation, question answering,
|
47 |
+
code generation, and more.
|
48 |
+
- source_sentence: What is one of the components mentioned alongside AI?
|
49 |
+
sentences:
|
50 |
+
- '2. Search Engines and Semantic Search
|
51 |
|
52 |
+
Traditional keyword-based search systems are being enhanced or replaced by semantic
|
53 |
+
search, where LLMs understand the meaning behind queries. Instead of just matching
|
54 |
+
words, they interpret intent.'
|
55 |
+
- For example, e-commerce websites can deploy LLM-powered assistants to help customers
|
56 |
+
find products, track orders, or get personalized recommendations—much more effectively
|
57 |
+
than traditional rule-based bots.
|
58 |
+
- Introduction to AI, Machine Learning, LLMs, and Their Integration
|
59 |
+
- source_sentence: What is required to provide intelligent features within broader
|
60 |
+
applications?
|
61 |
+
sentences:
|
62 |
+
- . For instance, a spam filter doesn’t just block emails with specific keywords—it
|
63 |
+
learns from thousands of examples what spam typically looks like.
|
64 |
+
- 'The Rise of LLM Integrations
|
65 |
|
66 |
+
While LLMs are powerful on their own, their true potential is unlocked through
|
67 |
+
integration—connecting these models with other software, services, or systems
|
68 |
+
to provide intelligent features within broader applications.
|
69 |
|
70 |
|
71 |
+
Here are some key ways LLMs are being integrated into the digital world:'
|
72 |
+
- For instance, in a document management system, a user might type "policies about
|
73 |
+
sick leave", and the system—integrated with an LLM—could retrieve documents discussing
|
74 |
+
"medical leave", "employee absence", and "illness policies", even if those exact
|
75 |
+
words weren’t used.
|
76 |
+
- source_sentence: What type of dialogues can LLMs simulate?
|
77 |
+
sentences:
|
78 |
+
- Companies are also experimenting with Retrieval-Augmented Generation (RAG)—a technique
|
79 |
+
where LLMs are paired with document databases (e.g., vector stores like Supabase,
|
80 |
+
Pinecone, or Weaviate) to answer questions with enterprise-specific knowledge.
|
81 |
+
- . For example, integrating an LLM into a customer support chatbot might involve
|
82 |
+
connecting it to a company’s internal knowledge base, enabling it to answer customer
|
83 |
+
questions using accurate, up-to-date information.
|
84 |
- '5. Education and Learning Platforms
|
85 |
|
86 |
Educational tools like Khanmigo (from Khan Academy) and other tutoring platforms
|
|
|
115 |
type: dim_768
|
116 |
metrics:
|
117 |
- type: cosine_accuracy@1
|
118 |
+
value: 0.6666666666666666
|
119 |
name: Cosine Accuracy@1
|
120 |
- type: cosine_accuracy@3
|
121 |
+
value: 0.8
|
122 |
name: Cosine Accuracy@3
|
123 |
- type: cosine_accuracy@5
|
124 |
+
value: 1.0
|
125 |
name: Cosine Accuracy@5
|
126 |
- type: cosine_accuracy@10
|
127 |
value: 1.0
|
128 |
name: Cosine Accuracy@10
|
129 |
- type: cosine_precision@1
|
130 |
+
value: 0.6666666666666666
|
131 |
name: Cosine Precision@1
|
132 |
- type: cosine_precision@3
|
133 |
+
value: 0.2666666666666667
|
134 |
name: Cosine Precision@3
|
135 |
- type: cosine_precision@5
|
136 |
+
value: 0.20000000000000007
|
137 |
name: Cosine Precision@5
|
138 |
- type: cosine_precision@10
|
139 |
value: 0.10000000000000003
|
140 |
name: Cosine Precision@10
|
141 |
- type: cosine_recall@1
|
142 |
+
value: 0.6666666666666666
|
143 |
name: Cosine Recall@1
|
144 |
- type: cosine_recall@3
|
145 |
+
value: 0.8
|
146 |
name: Cosine Recall@3
|
147 |
- type: cosine_recall@5
|
148 |
+
value: 1.0
|
149 |
name: Cosine Recall@5
|
150 |
- type: cosine_recall@10
|
151 |
value: 1.0
|
152 |
name: Cosine Recall@10
|
153 |
- type: cosine_ndcg@10
|
154 |
+
value: 0.8310827786456928
|
155 |
name: Cosine Ndcg@10
|
156 |
- type: cosine_mrr@10
|
157 |
+
value: 0.7766666666666667
|
158 |
name: Cosine Mrr@10
|
159 |
- type: cosine_map@100
|
160 |
+
value: 0.7766666666666667
|
161 |
name: Cosine Map@100
|
162 |
- task:
|
163 |
type: information-retrieval
|
|
|
173 |
value: 0.8
|
174 |
name: Cosine Accuracy@3
|
175 |
- type: cosine_accuracy@5
|
176 |
+
value: 0.8666666666666667
|
177 |
name: Cosine Accuracy@5
|
178 |
- type: cosine_accuracy@10
|
179 |
+
value: 1.0
|
180 |
name: Cosine Accuracy@10
|
181 |
- type: cosine_precision@1
|
182 |
value: 0.6666666666666666
|
|
|
185 |
value: 0.2666666666666667
|
186 |
name: Cosine Precision@3
|
187 |
- type: cosine_precision@5
|
188 |
+
value: 0.17333333333333337
|
189 |
name: Cosine Precision@5
|
190 |
- type: cosine_precision@10
|
191 |
+
value: 0.10000000000000003
|
192 |
name: Cosine Precision@10
|
193 |
- type: cosine_recall@1
|
194 |
value: 0.6666666666666666
|
|
|
197 |
value: 0.8
|
198 |
name: Cosine Recall@3
|
199 |
- type: cosine_recall@5
|
200 |
+
value: 0.8666666666666667
|
201 |
name: Cosine Recall@5
|
202 |
- type: cosine_recall@10
|
203 |
+
value: 1.0
|
204 |
name: Cosine Recall@10
|
205 |
- type: cosine_ndcg@10
|
206 |
+
value: 0.8203966331432972
|
207 |
name: Cosine Ndcg@10
|
208 |
- type: cosine_mrr@10
|
209 |
+
value: 0.7651851851851852
|
210 |
name: Cosine Mrr@10
|
211 |
- type: cosine_map@100
|
212 |
+
value: 0.7651851851851852
|
213 |
name: Cosine Map@100
|
214 |
- task:
|
215 |
type: information-retrieval
|
|
|
222 |
value: 0.6666666666666666
|
223 |
name: Cosine Accuracy@1
|
224 |
- type: cosine_accuracy@3
|
225 |
+
value: 0.8666666666666667
|
226 |
name: Cosine Accuracy@3
|
227 |
- type: cosine_accuracy@5
|
228 |
+
value: 1.0
|
229 |
name: Cosine Accuracy@5
|
230 |
- type: cosine_accuracy@10
|
231 |
value: 1.0
|
|
|
234 |
value: 0.6666666666666666
|
235 |
name: Cosine Precision@1
|
236 |
- type: cosine_precision@3
|
237 |
+
value: 0.28888888888888886
|
238 |
name: Cosine Precision@3
|
239 |
- type: cosine_precision@5
|
240 |
+
value: 0.20000000000000007
|
241 |
name: Cosine Precision@5
|
242 |
- type: cosine_precision@10
|
243 |
value: 0.10000000000000003
|
|
|
246 |
value: 0.6666666666666666
|
247 |
name: Cosine Recall@1
|
248 |
- type: cosine_recall@3
|
249 |
+
value: 0.8666666666666667
|
250 |
name: Cosine Recall@3
|
251 |
- type: cosine_recall@5
|
252 |
+
value: 1.0
|
253 |
name: Cosine Recall@5
|
254 |
- type: cosine_recall@10
|
255 |
value: 1.0
|
256 |
name: Cosine Recall@10
|
257 |
- type: cosine_ndcg@10
|
258 |
+
value: 0.8357043414408
|
259 |
name: Cosine Ndcg@10
|
260 |
- type: cosine_mrr@10
|
261 |
+
value: 0.7822222222222223
|
262 |
name: Cosine Mrr@10
|
263 |
- type: cosine_map@100
|
264 |
+
value: 0.7822222222222223
|
265 |
name: Cosine Map@100
|
266 |
- task:
|
267 |
type: information-retrieval
|
|
|
271 |
type: dim_128
|
272 |
metrics:
|
273 |
- type: cosine_accuracy@1
|
274 |
+
value: 0.5333333333333333
|
275 |
name: Cosine Accuracy@1
|
276 |
- type: cosine_accuracy@3
|
277 |
+
value: 0.7333333333333333
|
278 |
name: Cosine Accuracy@3
|
279 |
- type: cosine_accuracy@5
|
280 |
value: 0.8
|
281 |
name: Cosine Accuracy@5
|
282 |
- type: cosine_accuracy@10
|
283 |
+
value: 0.9333333333333333
|
284 |
name: Cosine Accuracy@10
|
285 |
- type: cosine_precision@1
|
286 |
+
value: 0.5333333333333333
|
287 |
name: Cosine Precision@1
|
288 |
- type: cosine_precision@3
|
289 |
+
value: 0.2444444444444445
|
290 |
name: Cosine Precision@3
|
291 |
- type: cosine_precision@5
|
292 |
value: 0.16000000000000003
|
293 |
name: Cosine Precision@5
|
294 |
- type: cosine_precision@10
|
295 |
+
value: 0.09333333333333335
|
296 |
name: Cosine Precision@10
|
297 |
- type: cosine_recall@1
|
298 |
+
value: 0.5333333333333333
|
299 |
name: Cosine Recall@1
|
300 |
- type: cosine_recall@3
|
301 |
+
value: 0.7333333333333333
|
302 |
name: Cosine Recall@3
|
303 |
- type: cosine_recall@5
|
304 |
value: 0.8
|
305 |
name: Cosine Recall@5
|
306 |
- type: cosine_recall@10
|
307 |
+
value: 0.9333333333333333
|
308 |
name: Cosine Recall@10
|
309 |
- type: cosine_ndcg@10
|
310 |
+
value: 0.7203966331432973
|
311 |
name: Cosine Ndcg@10
|
312 |
- type: cosine_mrr@10
|
313 |
+
value: 0.6540740740740741
|
314 |
name: Cosine Mrr@10
|
315 |
- type: cosine_map@100
|
316 |
+
value: 0.6592022792022793
|
317 |
name: Cosine Map@100
|
318 |
- task:
|
319 |
type: information-retrieval
|
|
|
323 |
type: dim_64
|
324 |
metrics:
|
325 |
- type: cosine_accuracy@1
|
326 |
+
value: 0.4666666666666667
|
327 |
name: Cosine Accuracy@1
|
328 |
- type: cosine_accuracy@3
|
329 |
+
value: 0.6666666666666666
|
330 |
name: Cosine Accuracy@3
|
331 |
- type: cosine_accuracy@5
|
332 |
value: 0.8
|
333 |
name: Cosine Accuracy@5
|
334 |
- type: cosine_accuracy@10
|
335 |
+
value: 0.8666666666666667
|
336 |
name: Cosine Accuracy@10
|
337 |
- type: cosine_precision@1
|
338 |
+
value: 0.4666666666666667
|
339 |
name: Cosine Precision@1
|
340 |
- type: cosine_precision@3
|
341 |
+
value: 0.22222222222222224
|
342 |
name: Cosine Precision@3
|
343 |
- type: cosine_precision@5
|
344 |
value: 0.16000000000000003
|
345 |
name: Cosine Precision@5
|
346 |
- type: cosine_precision@10
|
347 |
+
value: 0.08666666666666668
|
348 |
name: Cosine Precision@10
|
349 |
- type: cosine_recall@1
|
350 |
+
value: 0.4666666666666667
|
351 |
name: Cosine Recall@1
|
352 |
- type: cosine_recall@3
|
353 |
+
value: 0.6666666666666666
|
354 |
name: Cosine Recall@3
|
355 |
- type: cosine_recall@5
|
356 |
value: 0.8
|
357 |
name: Cosine Recall@5
|
358 |
- type: cosine_recall@10
|
359 |
+
value: 0.8666666666666667
|
360 |
name: Cosine Recall@10
|
361 |
- type: cosine_ndcg@10
|
362 |
+
value: 0.6507228370099043
|
363 |
name: Cosine Ndcg@10
|
364 |
- type: cosine_mrr@10
|
365 |
+
value: 0.5822222222222223
|
366 |
name: Cosine Mrr@10
|
367 |
- type: cosine_map@100
|
368 |
+
value: 0.58890559732665
|
369 |
name: Cosine Map@100
|
370 |
---
|
371 |
|
|
|
421 |
sentences = [
|
422 |
'What type of dialogues can LLMs simulate?',
|
423 |
'5. Education and Learning Platforms\nEducational tools like Khanmigo (from Khan Academy) and other tutoring platforms are leveraging LLMs to provide real-time help to students. LLMs can break down complex topics, provide feedback on writing, and simulate Socratic-style dialogues.',
|
424 |
+
'. For example, integrating an LLM into a customer support chatbot might involve connecting it to a company’s internal knowledge base, enabling it to answer customer questions using accurate, up-to-date information.',
|
425 |
]
|
426 |
embeddings = model.encode(sentences)
|
427 |
print(embeddings.shape)
|
|
|
468 |
|
469 |
| Metric | dim_768 | dim_512 | dim_256 | dim_128 | dim_64 |
|
470 |
|:--------------------|:-----------|:-----------|:-----------|:-----------|:-----------|
|
471 |
+
| cosine_accuracy@1 | 0.6667 | 0.6667 | 0.6667 | 0.5333 | 0.4667 |
|
472 |
+
| cosine_accuracy@3 | 0.8 | 0.8 | 0.8667 | 0.7333 | 0.6667 |
|
473 |
+
| cosine_accuracy@5 | 1.0 | 0.8667 | 1.0 | 0.8 | 0.8 |
|
474 |
+
| cosine_accuracy@10 | 1.0 | 1.0 | 1.0 | 0.9333 | 0.8667 |
|
475 |
+
| cosine_precision@1 | 0.6667 | 0.6667 | 0.6667 | 0.5333 | 0.4667 |
|
476 |
+
| cosine_precision@3 | 0.2667 | 0.2667 | 0.2889 | 0.2444 | 0.2222 |
|
477 |
+
| cosine_precision@5 | 0.2 | 0.1733 | 0.2 | 0.16 | 0.16 |
|
478 |
+
| cosine_precision@10 | 0.1 | 0.1 | 0.1 | 0.0933 | 0.0867 |
|
479 |
+
| cosine_recall@1 | 0.6667 | 0.6667 | 0.6667 | 0.5333 | 0.4667 |
|
480 |
+
| cosine_recall@3 | 0.8 | 0.8 | 0.8667 | 0.7333 | 0.6667 |
|
481 |
+
| cosine_recall@5 | 1.0 | 0.8667 | 1.0 | 0.8 | 0.8 |
|
482 |
+
| cosine_recall@10 | 1.0 | 1.0 | 1.0 | 0.9333 | 0.8667 |
|
483 |
+
| **cosine_ndcg@10** | **0.8311** | **0.8204** | **0.8357** | **0.7204** | **0.6507** |
|
484 |
+
| cosine_mrr@10 | 0.7767 | 0.7652 | 0.7822 | 0.6541 | 0.5822 |
|
485 |
+
| cosine_map@100 | 0.7767 | 0.7652 | 0.7822 | 0.6592 | 0.5889 |
|
486 |
|
487 |
<!--
|
488 |
## Bias, Risks and Limitations
|
|
|
505 |
* Size: 127 training samples
|
506 |
* Columns: <code>anchor</code> and <code>positive</code>
|
507 |
* Approximate statistics based on the first 127 samples:
|
508 |
+
| | anchor | positive |
|
509 |
+
|:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|
|
510 |
+
| type | string | string |
|
511 |
+
| details | <ul><li>min: 8 tokens</li><li>mean: 13.28 tokens</li><li>max: 25 tokens</li></ul> | <ul><li>min: 13 tokens</li><li>mean: 53.34 tokens</li><li>max: 86 tokens</li></ul> |
|
512 |
* Samples:
|
513 |
+
| anchor | positive |
|
514 |
+
|:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
515 |
+
| <code>What task mentioned is related to providing answers to inquiries?</code> | <code>. These include text generation, summarization, translation, question answering, code generation, and more.</code> |
|
516 |
+
| <code>What do LLMs learn to work effectively?</code> | <code>LLMs work by learning statistical relationships between words and phrases, allowing them to predict and generate language that feels natural. The power of these models lies not only in their size but also in the diversity of tasks they can perform with little to no task-specific training</code> |
|
517 |
+
| <code>In which industries is the generalization ability considered useful?</code> | <code>. This generalization ability makes them incredibly useful across industries—from customer service and education to software development and healthcare.</code> |
|
518 |
* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
|
519 |
```json
|
520 |
{
|
|
|
673 |
</details>
|
674 |
|
675 |
### Training Logs
|
676 |
+
| Epoch | Step | Training Loss | dim_768_cosine_ndcg@10 | dim_512_cosine_ndcg@10 | dim_256_cosine_ndcg@10 | dim_128_cosine_ndcg@10 | dim_64_cosine_ndcg@10 |
|
677 |
+
|:-------:|:------:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|
|
678 |
+
| 1.0 | 4 | - | 0.7790 | 0.7120 | 0.7474 | 0.6321 | 0.5684 |
|
679 |
+
| 2.0 | 8 | - | 0.8275 | 0.7966 | 0.8091 | 0.6904 | 0.6102 |
|
680 |
+
| 2.5 | 10 | 13.4453 | - | - | - | - | - |
|
681 |
+
| 3.0 | 12 | - | 0.8311 | 0.8204 | 0.8357 | 0.7178 | 0.6557 |
|
682 |
+
| **4.0** | **16** | **-** | **0.8311** | **0.8204** | **0.8357** | **0.7204** | **0.6507** |
|
683 |
|
684 |
* The bold row denotes the saved checkpoint.
|
685 |
|
model.safetensors
CHANGED
@@ -1,3 +1,3 @@
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
-
oid sha256:
|
3 |
size 596070136
|
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:b3d2c21c46d543a74109f26cb657b70ff5920256bc2c79d47d87b9086e1fae84
|
3 |
size 596070136
|