LLAMAPIE: Proactive In-Ear Conversation Assistants
Paper
•
2505.04066
•
Published
•
4
r PRO
oceansweep
AI & ML interests
None yet
Recent Activity
upvoted
a
paper
about 4 hours ago
LLAMAPIE: Proactive In-Ear Conversation Assistants
upvoted
a
paper
about 4 hours ago
Position: AI Competitions Provide the Gold Standard for Empirical Rigor
in GenAI Evaluation
upvoted
a
paper
about 4 hours ago
WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional
Websites from Scratch
Organizations
None yet
oceansweep's activity
upvoted
3
papers
about 4 hours ago
reacted to
codys12's
post with 👀
about 9 hours ago
Post
705
Introducing bitnet-r1-llama-8b and bitnet-r1-qwen-32b preview! These models are the first successful sub 1-billion-token finetune to BitNet architecture. We discovered that by adding an aditional input RMSNorm to each linear, you can finetune directly to BitNet with fast convergence to original model performance!
We are working on a pull request to use this extra RMS for any model.
To test these models now, install this fork of transformers:
Then load the models and test:
bitnet-r1-llama-8b and bitnet-r1-llama-32b were trained on ~ 300M and 200M tokens of the open-thoughts/OpenThoughts-114k dataset respectively, and were still significantly improving at the end of training. This preview simply demonstrates that the concept works, for future training runs we will leave the lm_head unquantized and align the last hidden state with the original model.
Huge thanks to the team that made this possible:
Gavin Childress, Aaron Herbst, Gavin Jones, Jasdeep Singh, Eli Vang, and Keagan Weinstock from the MSOE AI Club.
We are working on a pull request to use this extra RMS for any model.
To test these models now, install this fork of transformers:
pip install git+https://github.com/Codys12/transformers.gitThen load the models and test:
from transformers import (AutoModelForCausalLM, AutoTokenizer)
model_id = "codys12/bitnet-r1-qwen-32b"
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="cuda",
)
tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side="left")bitnet-r1-llama-8b and bitnet-r1-llama-32b were trained on ~ 300M and 200M tokens of the open-thoughts/OpenThoughts-114k dataset respectively, and were still significantly improving at the end of training. This preview simply demonstrates that the concept works, for future training runs we will leave the lm_head unquantized and align the last hidden state with the original model.
Huge thanks to the team that made this possible:
Gavin Childress, Aaron Herbst, Gavin Jones, Jasdeep Singh, Eli Vang, and Keagan Weinstock from the MSOE AI Club.
liked
a
model
1 day ago
upvoted
an
article
1 day ago
Article
Vision Language Models (Better, Faster, Stronger)
•
208
liked
a
model
2 days ago
liked
a
model
3 days ago
liked
a
model
4 days ago
upvoted
a
paper
4 days ago
liked
a
model
4 days ago
upvoted
5
papers
5 days ago
LLM-Independent Adaptive RAG: Let the Question Speak for Itself
Paper
•
2505.04253
•
Published
•
11
OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning
Paper
•
2505.04601
•
Published
•
18
OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolution
Paper
•
2505.04606
•
Published
•
6
Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities
Paper
•
2505.02567
•
Published
•
67
ZeroSearch: Incentivize the Search Capability of LLMs without Searching
Paper
•
2505.04588
•
Published
•
57
upvoted
2
papers
7 days ago
liked
a
model
7 days ago
liked
a
model
13 days ago
