open-thoughts/OpenThinker-7B
Text Generation
ā¢
Updated
ā¢
163
ā¢
21
R T
dingo-actual
AI & ML interests
None yet
Recent Activity
liked
a model
about 21 hours ago
open-thoughts/OpenThinker-7B
liked
a dataset
about 21 hours ago
open-thoughts/OpenThoughts-114k
liked
a dataset
1 day ago
Intel/orca_dpo_pairs
Organizations
dingo-actual's activity
liked
a
model
about 21 hours ago
liked
a
dataset
about 21 hours ago
liked
a
dataset
1 day ago
liked
a
model
1 day ago
liked
a
model
5 days ago
reacted to
lewtun's
post with š„
5 days ago
Post
9323
We are reproducing the full DeepSeek R1 data and training pipeline so everybody can use their recipe. Instead of doing it in secret we can do it together in the open!
š§Ŗ Step 1: replicate the R1-Distill models by distilling a high-quality reasoning corpus from DeepSeek-R1.
š§ Step 2: replicate the pure RL pipeline that DeepSeek used to create R1-Zero. This will involve curating new, large-scale datasets for math, reasoning, and code.
š„ Step 3: show we can go from base model -> SFT -> RL via multi-stage training.
Follow along: https://github.com/huggingface/open-r1
š§Ŗ Step 1: replicate the R1-Distill models by distilling a high-quality reasoning corpus from DeepSeek-R1.
š§ Step 2: replicate the pure RL pipeline that DeepSeek used to create R1-Zero. This will involve curating new, large-scale datasets for math, reasoning, and code.
š„ Step 3: show we can go from base model -> SFT -> RL via multi-stage training.
Follow along: https://github.com/huggingface/open-r1
- 5 replies
replied to
singhsidhukuldeep's
post
5 days ago
reacted to
singhsidhukuldeep's
post with š
5 days ago
Post
562
Exciting breakthrough in Text Embeddings: Introducing LENS (Lexicon-based EmbeddiNgS)!
A team of researchers from University of Amsterdam, University of Technology Sydney, and Tencent have developed a groundbreaking approach that outperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB).
>> Key Technical Innovations:
- LENS consolidates vocabulary space through token embedding clustering, addressing the inherent redundancy in LLM tokenizers
- Implements bidirectional attention and innovative pooling strategies to unlock the full potential of LLMs
- Each dimension corresponds to token clusters instead of individual tokens, creating more coherent and compact embeddings
- Achieves competitive performance with just 4,000-8,000 dimensional embeddings, matching the size of dense counterparts
>> Under the Hood:
The framework applies KMeans clustering to token embeddings from the language modeling head, replacing original embeddings with cluster centroids. This reduces dimensionality while preserving semantic relationships.
>> Results:
- Outperforms dense embeddings on MTEB benchmark
- Achieves state-of-the-art performance when combined with dense embeddings on BEIR retrieval tasks
- Demonstrates superior performance across clustering, classification, and retrieval tasks
This work opens new possibilities for more efficient and interpretable text embeddings. The code will be available soon.
A team of researchers from University of Amsterdam, University of Technology Sydney, and Tencent have developed a groundbreaking approach that outperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB).
>> Key Technical Innovations:
- LENS consolidates vocabulary space through token embedding clustering, addressing the inherent redundancy in LLM tokenizers
- Implements bidirectional attention and innovative pooling strategies to unlock the full potential of LLMs
- Each dimension corresponds to token clusters instead of individual tokens, creating more coherent and compact embeddings
- Achieves competitive performance with just 4,000-8,000 dimensional embeddings, matching the size of dense counterparts
>> Under the Hood:
The framework applies KMeans clustering to token embeddings from the language modeling head, replacing original embeddings with cluster centroids. This reduces dimensionality while preserving semantic relationships.
>> Results:
- Outperforms dense embeddings on MTEB benchmark
- Achieves state-of-the-art performance when combined with dense embeddings on BEIR retrieval tasks
- Demonstrates superior performance across clustering, classification, and retrieval tasks
This work opens new possibilities for more efficient and interpretable text embeddings. The code will be available soon.
-
- 1 reply
liked
2
models
6 days ago
reacted to
burtenshaw's
post with š¤Æ
6 days ago
Post
2059
AI was built on side projects!
liked
a
model
7 days ago
liked
3
models
8 days ago
liked
a
model
9 days ago
liked
4
models
10 days ago
