datasets:
- tiiuae/falcon-refinedweb
language:
- en
library_name: transformers.js
license: mit
pipeline_tag: feature-extraction
base_model:
- chandar-lab/NeoBERT
NeoBERT
NeoBERT is a next-generation encoder model for English text representation, pre-trained from scratch on the RefinedWeb dataset. NeoBERT integrates state-of-the-art advancements in architecture, modern data, and optimized pre-training methodologies. It is designed for seamless adoption: it serves as a plug-and-play replacement for existing base models, relies on an optimal depth-to-width ratio, and leverages an extended context length of 4,096 tokens. Despite its compact 250M parameter footprint, it is the most efficient model of its kind and achieves state-of-the-art results on the massive MTEB benchmark, outperforming BERT large, RoBERTa large, NomicBERT, and ModernBERT under identical fine-tuning conditions.
Usage
ONNXRuntime
from transformers import AutoTokenizer
from huggingface_hub import hf_hub_download
import onnxruntime as ort
model_id = "onnx-community/NeoBERT-ONNX"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model_file = hf_hub_download(model_id, filename="onnx/model.onnx")
session = ort.InferenceSession(model_file)
text = ["NeoBERT is the most efficient model of its kind!"]
inputs = tokenizer(text, return_tensors="np").data
outputs = session.run(None, inputs)[0]
embeddings = outputs[:, 0, :]
print(f"{embeddings.shape=}") # (1, 768)
Conversion
The export script can be found at ./export.py.