๐ฎ๐น๐ฏ๐ต๐ง๐ท Generating multilingual instruction datasets with Magpie ๐ฆโโฌ
โข
18
Stefano Fiorucci PRO
anakin87
ยท
AI & ML interests
Contributing to Haystack LLM framework ๐๏ธ. Language Models: orchestration, post-training, synthetic data...
Recent Activity
updated
a Space
1 day ago
anakin87/fact-checking-rocks
updated
a Space
1 day ago
anakin87/Phi-3.5-mini-ITA
updated
a Space
1 day ago
anakin87/gemma-2-2b-neogenesis-ita
Articles
Organizations
anakin87's activity
updated
4
Spaces
1 day ago
updated
a
model
1 day ago
liked
a
dataset
1 day ago
updated
a
model
1 day ago
liked
a
model
1 day ago
published
a
Space
1 day ago
updated
a
collection
1 day ago
upvoted
a
collection
3 days ago
liked
a
Space
3 days ago
reacted to
tomaarsen's
post with โค๏ธ
3 days ago
Post
4090
๐๏ธ Today I'm introducing a method to train static embedding models that run 100x to 400x faster on CPU than common embedding models, while retaining 85%+ of the quality! Including 2 fully open models: training scripts, datasets, metrics.
We apply our recipe to train 2 Static Embedding models that we release today! We release:
2๏ธโฃ an English Retrieval model and a general-purpose Multilingual similarity model (e.g. classification, clustering, etc.), both Apache 2.0
๐ง my modern training strategy: ideation -> dataset choice -> implementation -> evaluation
๐ my training scripts, using the Sentence Transformers library
๐ my Weights & Biases reports with losses & metrics
๐ my list of 30 training and 13 evaluation datasets
The 2 Static Embedding models have the following properties:
๐๏ธ Extremely fast, e.g. 107500 sentences per second on a consumer CPU, compared to 270 for 'all-mpnet-base-v2' and 56 for 'gte-large-en-v1.5'
0๏ธโฃ Zero active parameters: No Transformer blocks, no attention, not even a matrix multiplication. Super speed!
๐ No maximum sequence length! Embed texts at any length (note: longer texts may embed worse)
๐ Linear instead of exponential complexity: 2x longer text takes 2x longer, instead of 2.5x or more.
๐ช Matryoshka support: allow you to truncate embeddings with minimal performance loss (e.g. 4x smaller with a 0.56% perf. decrease for English Similarity tasks)
Check out the full blogpost if you'd like to 1) use these lightning-fast models or 2) learn how to train them with consumer-level hardware: https://huggingface.co/blog/static-embeddings
The blogpost contains a lengthy list of possible advancements; I'm very confident that our 2 models are only the tip of the iceberg, and we may be able to get even better performance.
Alternatively, check out the models:
* sentence-transformers/static-retrieval-mrl-en-v1
* sentence-transformers/static-similarity-mrl-multilingual-v1
We apply our recipe to train 2 Static Embedding models that we release today! We release:
2๏ธโฃ an English Retrieval model and a general-purpose Multilingual similarity model (e.g. classification, clustering, etc.), both Apache 2.0
๐ง my modern training strategy: ideation -> dataset choice -> implementation -> evaluation
๐ my training scripts, using the Sentence Transformers library
๐ my Weights & Biases reports with losses & metrics
๐ my list of 30 training and 13 evaluation datasets
The 2 Static Embedding models have the following properties:
๐๏ธ Extremely fast, e.g. 107500 sentences per second on a consumer CPU, compared to 270 for 'all-mpnet-base-v2' and 56 for 'gte-large-en-v1.5'
0๏ธโฃ Zero active parameters: No Transformer blocks, no attention, not even a matrix multiplication. Super speed!
๐ No maximum sequence length! Embed texts at any length (note: longer texts may embed worse)
๐ Linear instead of exponential complexity: 2x longer text takes 2x longer, instead of 2.5x or more.
๐ช Matryoshka support: allow you to truncate embeddings with minimal performance loss (e.g. 4x smaller with a 0.56% perf. decrease for English Similarity tasks)
Check out the full blogpost if you'd like to 1) use these lightning-fast models or 2) learn how to train them with consumer-level hardware: https://huggingface.co/blog/static-embeddings
The blogpost contains a lengthy list of possible advancements; I'm very confident that our 2 models are only the tip of the iceberg, and we may be able to get even better performance.
Alternatively, check out the models:
* sentence-transformers/static-retrieval-mrl-en-v1
* sentence-transformers/static-similarity-mrl-multilingual-v1
- 1 reply
posted
an
update
3 days ago
Post
452
Hey, it has been a while... I was busy participating in ๐ ๐๐๐ฆ๐ฆ๐ ๐๐จ๐ฆ๐ฉ๐๐ญ๐ข๐ญ๐ข๐จ๐ง!
Here's the idea: Gemma open models have a large vocabulary size (256K), so improving them for a specific language or cultural context should be pretty affordable - no need for continued pre-training.
My submission: ๐๐๐ฎ๐น ๐๐๐จ๐ ๐๐ง๐๐ฌ๐ข๐ฌ - ๐๐จ๐ฌ๐ญ-๐๐ซ๐๐ข๐ง๐ข๐ง๐ ๐๐๐ฆ๐ฆ๐ ๐๐จ๐ซ ๐๐ญ๐๐ฅ๐ข๐๐ง ๐๐ง๐ ๐๐๐ฒ๐จ๐ง๐
๐ Kaggle notebook: https://www.kaggle.com/code/anakin87/post-training-gemma-for-italian-and-beyond
In this notebook, I show how I improve the performance of Gemma 2 2B on Italian via Post-Training.
I believe this method is adaptable to other languages and model sizes.
๐๐ฆ๐บ ๐๐ต๐ฆ๐ฑ๐ด
๐ Choose reference metrics
๐งโ๐ฌ Data curation for Instruction Fine Tuning: identify existing datasets + generate synthetic data
๐๏ธโโ๏ธ Efficient Instruction Fine Tuning with Spectrum
๐งโ๐ฌ Data curation for Preference Tuning: identify existing datasets + generate synthetic data
๐๐ Efficient Direct Preference Optimization with Spectrum
๐ Evaluation
๐ค Hugging Face collection (with models and datasets): anakin87/gemma-neogenesis-67824b7bf13ac9cfe091fe2e
I'm also planning a ๐ Gemma Giveaway (on LinkedIn - https://www.linkedin.com/in/stefano-fiorucci) in the next few days - sharing techniques, datasets, and models I used for my project... so stay tuned! ๐ป
Here's the idea: Gemma open models have a large vocabulary size (256K), so improving them for a specific language or cultural context should be pretty affordable - no need for continued pre-training.
My submission: ๐๐๐ฎ๐น ๐๐๐จ๐ ๐๐ง๐๐ฌ๐ข๐ฌ - ๐๐จ๐ฌ๐ญ-๐๐ซ๐๐ข๐ง๐ข๐ง๐ ๐๐๐ฆ๐ฆ๐ ๐๐จ๐ซ ๐๐ญ๐๐ฅ๐ข๐๐ง ๐๐ง๐ ๐๐๐ฒ๐จ๐ง๐
๐ Kaggle notebook: https://www.kaggle.com/code/anakin87/post-training-gemma-for-italian-and-beyond
In this notebook, I show how I improve the performance of Gemma 2 2B on Italian via Post-Training.
I believe this method is adaptable to other languages and model sizes.
๐๐ฆ๐บ ๐๐ต๐ฆ๐ฑ๐ด
๐ Choose reference metrics
๐งโ๐ฌ Data curation for Instruction Fine Tuning: identify existing datasets + generate synthetic data
๐๏ธโโ๏ธ Efficient Instruction Fine Tuning with Spectrum
๐งโ๐ฌ Data curation for Preference Tuning: identify existing datasets + generate synthetic data
๐๐ Efficient Direct Preference Optimization with Spectrum
๐ Evaluation
๐ค Hugging Face collection (with models and datasets): anakin87/gemma-neogenesis-67824b7bf13ac9cfe091fe2e
I'm also planning a ๐ Gemma Giveaway (on LinkedIn - https://www.linkedin.com/in/stefano-fiorucci) in the next few days - sharing techniques, datasets, and models I used for my project... so stay tuned! ๐ป
liked
2
models
3 days ago
updated
a
dataset
4 days ago
updated
a
collection
4 days ago
liked
a
model
4 days ago
