Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning
Paper
•
2503.07572
•
Published
•
40
SomosNLP
non-profit
AI & ML interests
Democratizar el PLN en español e incentivar su aplicación para generar impacto social 💛
Recent Activity
View all activity
somosnlp's activity
lewtun
authored
a
paper
13 days ago
Post
2079
Introducing OlympicCoder: a series of open reasoning models that can solve olympiad-level programming problems 🧑💻
- 7B open-r1/OlympicCoder-7B
- 32B open-r1/OlympicCoder-32B
We find that OlympicCoder models outperform Claude 3.7 Sonnet, as well as others over 100x larger 💪
Together with the models, we are releasing:
📊CodeForces-CoTs: new dataset of code problems from the most popular competitive coding platform, with R1 traces in C++ and Python open-r1/codeforces-cots
🏆 IOI'2024: a new benchmark of VERY hard programming problems where even frontier models struggle to match human performance open-r1/ioi
For links to the models and datasets, check out our latest progress report from Open R1: https://huggingface.co/blog/open-r1/update-3
- 7B open-r1/OlympicCoder-7B
- 32B open-r1/OlympicCoder-32B
We find that OlympicCoder models outperform Claude 3.7 Sonnet, as well as others over 100x larger 💪
Together with the models, we are releasing:
📊CodeForces-CoTs: new dataset of code problems from the most popular competitive coding platform, with R1 traces in C++ and Python open-r1/codeforces-cots
🏆 IOI'2024: a new benchmark of VERY hard programming problems where even frontier models struggle to match human performance open-r1/ioi
For links to the models and datasets, check out our latest progress report from Open R1: https://huggingface.co/blog/open-r1/update-3
- 1 reply
davidberenstein1957
posted
an
update
19 days ago
Post
2370
🔥 Text2SQL, explore and share any data analysis!
🤗 Hugging Face - Dataset Studio is an amazing new feature.
🚀 Start yourself: fka/awesome-chatgpt-prompts
📺 YouTube: https://youtu.be/5LUZq7MHolA?feature=shared
🤗 Hugging Face - Dataset Studio is an amazing new feature.
🚀 Start yourself: fka/awesome-chatgpt-prompts
📺 YouTube: https://youtu.be/5LUZq7MHolA?feature=shared
davidberenstein1957
posted
an
update
21 days ago
Post
4218
🥊 Epic Agent Framework Showdown! Available today!
🔵 In the blue corner, the versatile challenger with a proven track record of knowledge retrieval: LlamaIndex!
🛑 In the red corner, the defender, weighing in with lightweight efficiency: Hugging Face smolagents!
🔗 URL: https://huggingface.co/agents-course
We just published the LlamaIndex unit for the agents course, and it is set to offer a great contrast between the smolagents unit by looking at
- What makes llama-index stand-out
- How the LlamaHub is used for integrations
- Creating QueryEngine components
- Using agents and tools
- Agentic and multi-agent workflows
The team has been working flat-out on this for a few weeks. Supported by Logan Markewich and Laurie Voss over at LlamaIndex.
Who won? You decide!
🔵 In the blue corner, the versatile challenger with a proven track record of knowledge retrieval: LlamaIndex!
🛑 In the red corner, the defender, weighing in with lightweight efficiency: Hugging Face smolagents!
🔗 URL: https://huggingface.co/agents-course
We just published the LlamaIndex unit for the agents course, and it is set to offer a great contrast between the smolagents unit by looking at
- What makes llama-index stand-out
- How the LlamaHub is used for integrations
- Creating QueryEngine components
- Using agents and tools
- Agentic and multi-agent workflows
The team has been working flat-out on this for a few weeks. Supported by Logan Markewich and Laurie Voss over at LlamaIndex.
Who won? You decide!
davidberenstein1957
posted
an
update
21 days ago
Post
3018
🫸 New release to push vector search to the Hub with vicinity and work with any serialisable objects.
🧑🏫 KNN, HNSW, USEARCH, ANNOY, PYNNDESCENT, FAISS, and VOYAGER.
🔗 Example Repo: minishlab/my-vicinity-repo
🧑🏫 KNN, HNSW, USEARCH, ANNOY, PYNNDESCENT, FAISS, and VOYAGER.
🔗 Example Repo: minishlab/my-vicinity-repo
davidberenstein1957
posted
an
update
about 1 month ago
Post
3295
🚀 Find banger tools for your smolagents!
I created the Tools gallery, which makes tools specifically developed by/for smolagents searchable and visible. This will help with:
- inspiration
- best practices
- finding cool tools
Space: davidberenstein1957/smolagents-and-tools
I created the Tools gallery, which makes tools specifically developed by/for smolagents searchable and visible. This will help with:
- inspiration
- best practices
- finding cool tools
Space: davidberenstein1957/smolagents-and-tools
- 1 reply
Post
4979
Introducing OpenR1-Math-220k!
open-r1/OpenR1-Math-220k
The community has been busy distilling DeepSeek-R1 from inference providers, but we decided to have a go at doing it ourselves from scratch 💪
What’s new compared to existing reasoning datasets?
♾ Based on AI-MO/NuminaMath-1.5: we focus on math reasoning traces and generate answers for problems in NuminaMath 1.5, an improved version of the popular NuminaMath-CoT dataset.
🐳 800k R1 reasoning traces: We generate two answers for 400k problems using DeepSeek R1. The filtered dataset contains 220k problems with correct reasoning traces.
📀 512 H100s running locally: Instead of relying on an API, we leverage vLLM and SGLang to run generations locally on our science cluster, generating 180k reasoning traces per day.
⏳ Automated filtering: We apply Math Verify to only retain problems with at least one correct answer. We also leverage Llama3.3-70B-Instruct as a judge to retrieve more correct examples (e.g for cases with malformed answers that can’t be verified with a rules-based parser)
📊 We match the performance of DeepSeek-Distill-Qwen-7B by finetuning Qwen-7B-Math-Instruct on our dataset.
🔎 Read our blog post for all the nitty gritty details: https://huggingface.co/blog/open-r1/update-2
open-r1/OpenR1-Math-220k
The community has been busy distilling DeepSeek-R1 from inference providers, but we decided to have a go at doing it ourselves from scratch 💪
What’s new compared to existing reasoning datasets?
♾ Based on AI-MO/NuminaMath-1.5: we focus on math reasoning traces and generate answers for problems in NuminaMath 1.5, an improved version of the popular NuminaMath-CoT dataset.
🐳 800k R1 reasoning traces: We generate two answers for 400k problems using DeepSeek R1. The filtered dataset contains 220k problems with correct reasoning traces.
📀 512 H100s running locally: Instead of relying on an API, we leverage vLLM and SGLang to run generations locally on our science cluster, generating 180k reasoning traces per day.
⏳ Automated filtering: We apply Math Verify to only retain problems with at least one correct answer. We also leverage Llama3.3-70B-Instruct as a judge to retrieve more correct examples (e.g for cases with malformed answers that can’t be verified with a rules-based parser)
📊 We match the performance of DeepSeek-Distill-Qwen-7B by finetuning Qwen-7B-Math-Instruct on our dataset.
🔎 Read our blog post for all the nitty gritty details: https://huggingface.co/blog/open-r1/update-2
davidberenstein1957
posted
an
update
about 1 month ago
Post
2479
Fine-tune Deepseek-R1 with a Synthetic Reasoning Dataset
Blog: https://huggingface.co/blog/sdiazlor/fine-tune-deepseek-with-a-synthetic-reasoning-data
Blog: https://huggingface.co/blog/sdiazlor/fine-tune-deepseek-with-a-synthetic-reasoning-data
plaguss
authored
a
paper
about 2 months ago
lewtun
authored
a
paper
about 2 months ago
davidberenstein1957
posted
an
update
about 2 months ago
Post
2075
Agentic RAG: Applied, visual, and step-by-step! 🐾
Get familiar with the Agents and tools, not the bells and whistles!
Retrieve - Augment and now GENERATE.
part 3: https://huggingface.co/blog/davidberenstein1957/ai-blueprint-agentic-rag-part-3-generate
Get familiar with the Agents and tools, not the bells and whistles!
Retrieve - Augment and now GENERATE.
part 3: https://huggingface.co/blog/davidberenstein1957/ai-blueprint-agentic-rag-part-3-generate
davidberenstein1957
posted
an
update
about 2 months ago
Post
2880
Anyone can create free hosted tools for their AI agents! 🔥
Agentic RAG stack part 2 - augment
Augment retrieval results by reranking optimises content without increasing time too much
part2: https://huggingface.co/blog/davidberenstein1957/ai-blueprint-agentic-rag-part-2-augment
code: https://github.com/huggingface/ai-blueprint
Agentic RAG stack part 2 - augment
Augment retrieval results by reranking optimises content without increasing time too much
part2: https://huggingface.co/blog/davidberenstein1957/ai-blueprint-agentic-rag-part-2-augment
code: https://github.com/huggingface/ai-blueprint
mariagrandury
updated
a
collection
about 2 months ago
davidberenstein1957
posted
an
update
about 2 months ago
Post
2011
Creating an agentic RAG stack on the Hugging Face Hub - part 1 - retrieval (1/5).
🚀 Web apps and microservices included!
Chunk, embed and index documents at a huge scale without overhead.
Blog: https://huggingface.co/blog/davidberenstein1957/ai-blueprint-agentic-rag-part-1-retrieve
🚀 Web apps and microservices included!
Chunk, embed and index documents at a huge scale without overhead.
Blog: https://huggingface.co/blog/davidberenstein1957/ai-blueprint-agentic-rag-part-1-retrieve
davidberenstein1957
posted
an
update
about 2 months ago
Post
1634
tldr; Parquet is awesome, DuckDB too!
Datasets on the Hugging Face Hub rely on parquet files. We can interact with these files using DuckDB as a fast in-memory database system. One of DuckDB’s features is vector similarity search which can be used with or without an index.
blog:
https://huggingface.co/learn/cookbook/vector_search_with_hub_as_backend
Datasets on the Hugging Face Hub rely on parquet files. We can interact with these files using DuckDB as a fast in-memory database system. One of DuckDB’s features is vector similarity search which can be used with or without an index.
blog:
https://huggingface.co/learn/cookbook/vector_search_with_hub_as_backend
mariagrandury
updated
2
collections
about 2 months ago
davidberenstein1957
posted
an
update
about 2 months ago
Post
1809
Let's uncover the post-training dataset from DeepSeek-R1 with Magpie!
Pass pre-query tokens
We can get realistic examples!
Gist: https://gist.github.com/davidberenstein1957/3f20046ce57395a6aba13f8b4e956b59
Pass pre-query tokens
<｜begin▁of▁sentence｜>User: , let the model generate the rest.We can get realistic examples!
Gist: https://gist.github.com/davidberenstein1957/3f20046ce57395a6aba13f8b4e956b59
-
- 6 replies
Post
10279
We are reproducing the full DeepSeek R1 data and training pipeline so everybody can use their recipe. Instead of doing it in secret we can do it together in the open!
🧪 Step 1: replicate the R1-Distill models by distilling a high-quality reasoning corpus from DeepSeek-R1.
🧠 Step 2: replicate the pure RL pipeline that DeepSeek used to create R1-Zero. This will involve curating new, large-scale datasets for math, reasoning, and code.
🔥 Step 3: show we can go from base model -> SFT -> RL via multi-stage training.
Follow along: https://github.com/huggingface/open-r1
🧪 Step 1: replicate the R1-Distill models by distilling a high-quality reasoning corpus from DeepSeek-R1.
🧠 Step 2: replicate the pure RL pipeline that DeepSeek used to create R1-Zero. This will involve curating new, large-scale datasets for math, reasoning, and code.
🔥 Step 3: show we can go from base model -> SFT -> RL via multi-stage training.
Follow along: https://github.com/huggingface/open-r1
- 5 replies
davidberenstein1957
posted
an
update
2 months ago
Post
1893
The RAG's in the bag!
You can now use the Synthetic Data Generator with your own domain-specific seed data to generate a dataset for fine-tuning retrieval or reranking models.
GitHub: https://buff.ly/49IDSmd
Spaces: https://buff.ly/3Y1S99z
Blog: https://huggingface.co/blog/sdiazlor/fine-tune-modernbert-for-rag-with-synthetic-data
You can now use the Synthetic Data Generator with your own domain-specific seed data to generate a dataset for fine-tuning retrieval or reranking models.
GitHub: https://buff.ly/49IDSmd
Spaces: https://buff.ly/3Y1S99z
Blog: https://huggingface.co/blog/sdiazlor/fine-tune-modernbert-for-rag-with-synthetic-data
- 1 reply