Runtime error
BandiCount
🐨
Gordon
GordonM
AI & ML interests
Data Science for good
Recent Activity
reacted
to
MoritzLaurer's
post
with 👍
about 2 months ago
Quite excited by the ModernBERT release! 0.15/0.4B small, 2T modern pre-training data and tokenizer with code, 8k context window, great efficient model for embeddings & classification!
This will probably be the basis for many future SOTA encoders! And I can finally stop using DeBERTav3 from 2021 :D
Congrats @answerdotai, @LightOnIO and collaborators like @tomaarsen !
Paper and models here 👇https://huggingface.co/collections/answerdotai/modernbert-67627ad707a4acbf33c41deb
reacted
to
MoritzLaurer's
post
with 🔥
about 2 months ago
🚀 Releasing a new zeroshot-classifier based on ModernBERT! Some key takeaways:
- ⚡ Speed & efficiency: It's multiple times faster and uses significantly less memory than DeBERTav3. You can use larger batch sizes and enabling bf16 (instead of fp16) gave me a ~2x speed boost as well
- 📉 Performance tradeoff: It performs slightly worse than DeBERTav3 on average across my zeroshot classification task collection
- 🧠 Use cases: I recommend using it for scenarios requiring speed and a larger context window (8k).
- 💡 What’s next? I’m preparing a newer version trained on better + longer synthetic data to fully leverage the 8k context window and improve upon the training mix of my older zeroshot-v2.0 models. I also hope that there will be a multilingual variant in the future.
Great work by https://huggingface.co/answerdotai !
If you’re looking for a high-speed zeroshot classifier, give it a try!
📄 Resources below: 👇
Base model: https://huggingface.co/MoritzLaurer/ModernBERT-base-zeroshot-v2.0
Large model: https://huggingface.co/MoritzLaurer/ModernBERT-large-zeroshot-v2.0
Updated zeroshot collection: https://huggingface.co/collections/MoritzLaurer/zeroshot-classifiers-6548b4ff407bb19ff5c3ad6f
ModernBERT collection with paper: https://huggingface.co/collections/answerdotai/modernbert-67627ad707a4acbf33c41deb
liked
a model
3 months ago
Varosa/SeamlessExpressive
Organizations
