|
|
--- |
|
|
language: multilingual |
|
|
license: mit |
|
|
tags: |
|
|
- onnx |
|
|
- optimum |
|
|
- quantized |
|
|
- int8 |
|
|
- text-embedding |
|
|
- onnxruntime |
|
|
- opset14 |
|
|
- text-classification |
|
|
- gpu |
|
|
- optimized |
|
|
datasets: |
|
|
- mmarco |
|
|
pipeline_tag: sentence-similarity |
|
|
--- |
|
|
|
|
|
# gte-multilingual-reranker-base-onnx-op14-opt-gpu-int8-quantized |
|
|
|
|
|
This model is a quantized ONNX version of [Alibaba-NLP/gte-multilingual-reranker-base](https://huggingface.co/Alibaba-NLP/gte-multilingual-reranker-base) using ONNX opset 14. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Quantization Type**: INT8 |
|
|
- **ONNX Opset**: 14 |
|
|
- **Task**: text-classification |
|
|
- **Target Device**: GPU |
|
|
- **Optimized**: Yes |
|
|
- **Framework**: ONNX Runtime |
|
|
- **Original Model**: [Alibaba-NLP/gte-multilingual-reranker-base](https://huggingface.co/Alibaba-NLP/gte-multilingual-reranker-base) |
|
|
- **Quantized On**: 2025-03-27 |
|
|
|
|
|
## Environment and Package Versions |
|
|
|
|
|
| Package | Version | |
|
|
| --- | --- | |
|
|
| transformers | 4.48.3 | |
|
|
| optimum | 1.24.0 | |
|
|
| onnx | 1.17.0 | |
|
|
| onnxruntime | 1.21.0 | |
|
|
| torch | 2.5.1 | |
|
|
| numpy | 1.26.4 | |
|
|
| huggingface_hub | 0.28.1 | |
|
|
| python | 3.12.9 | |
|
|
| system | Darwin 24.3.0 | |
|
|
|
|
|
|
|
|
### Applied Optimizations |
|
|
|
|
|
| Optimization | Setting | |
|
|
| --- | --- | |
|
|
| Graph Optimization Level | Extended | |
|
|
| Optimize for GPU | Yes | |
|
|
| Use FP16 | No | |
|
|
| Transformers Specific Optimizations Enabled | Yes | |
|
|
| Gelu Fusion Enabled | Yes | |
|
|
| Layer Norm Fusion Enabled | Yes | |
|
|
| Attention Fusion Enabled | Yes | |
|
|
| Skip Layer Norm Fusion Enabled | Yes | |
|
|
| Gelu Approximation Enabled | Yes | |
|
|
|
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from optimum.onnxruntime import ORTModelForSequenceClassification |
|
|
from transformers import AutoTokenizer |
|
|
|
|
|
# Load model and tokenizer |
|
|
model = ORTModelForSequenceClassification.from_pretrained("quantized_model") |
|
|
tokenizer = AutoTokenizer.from_pretrained("quantized_model") |
|
|
|
|
|
# Prepare input |
|
|
text = "Your text here" |
|
|
inputs = tokenizer(text, return_tensors="pt") |
|
|
|
|
|
# Run inference |
|
|
outputs = model(**inputs) |
|
|
``` |
|
|
|
|
|
## Quantization Process |
|
|
|
|
|
This model was quantized using ONNX Runtime with int8 quantization. |
|
|
The quantization was performed using the Optimum library from Hugging Face with opset 14. |
|
|
Graph optimization was applied during export, targeting GPU devices. |
|
|
|
|
|
|
|
|
## Performance Comparison |
|
|
|
|
|
Quantized models generally offer better inference speed with a slight trade-off in accuracy. |
|
|
This INT8 quantized model should provide significantly faster inference than the original model. |
|
|
|
