Create README.md

README.md

@@ -1,3 +1,16 @@
----
-license: mit
----
+---
+datasets:
+- tiiuae/falcon-refinedweb
+language:
+- en
+library_name: transformers.js
+license: mit
+pipeline_tag: feature-extraction
+---
+
+# NeoBERT
+
+NeoBERT is a **next-generation encoder** model for English text representation, pre-trained from scratch on the RefinedWeb dataset. NeoBERT integrates state-of-the-art advancements in architecture, modern data, and optimized pre-training methodologies. It is designed for seamless adoption: it serves as a plug-and-play replacement for existing base models, relies on an **optimal depth-to-width ratio**, and leverages an extended context length of **4,096 tokens**. Despite its compact 250M parameter footprint, it is the most efficient model of its kind and achieves **state-of-the-art results** on the massive MTEB benchmark, outperforming BERT large, RoBERTa large, NomicBERT, and ModernBERT under identical fine-tuning conditions. 
+
+- Paper: [paper](https://arxiv.org/abs/2502.19587)
+- Repository: [github](https://github.com/chandar-lab/NeoBERT).