|
|
--- |
|
|
language: |
|
|
- en |
|
|
- zh |
|
|
tags: |
|
|
- fp8 |
|
|
- quantization |
|
|
- dynamic |
|
|
- vision-language |
|
|
- multimodal |
|
|
- vLLM |
|
|
- llm-compressor |
|
|
- internvl3 |
|
|
pipeline_tag: image-text-to-text |
|
|
inference: false |
|
|
license: mit |
|
|
base_model: |
|
|
- OpenGVLab/InternVL3-8B |
|
|
--- |
|
|
|
|
|
# 🔥 InternVL3-8B-FP8-Dynamic: Optimized Vision-Language Model 🔥 |
|
|
This is a **FP8 dynamic quantized** version of [OpenGVLab/InternVL3-8B](https://huggingface.co/OpenGVLab/InternVL3-8B), optimized for high-performance inference with vLLM. |
|
|
The model utilizes **dynamic FP8 quantization** for optimal ease of use and deployment, achieving significant speedup with minimal accuracy degradation on vision-language tasks. |
|
|
|
|
|
## 🔧 Usage |
|
|
### With vLLM (Recommended) |
|
|
```python |
|
|
from vllm import LLM, SamplingParams |
|
|
|
|
|
# Load the quantized model |
|
|
model = LLM( |
|
|
model="brandonbeiler/InternVL3-8B-FP8-Dynamic", |
|
|
trust_remote_code=True, |
|
|
max_model_len=8192, |
|
|
tensor_parallel_size=1, # Adjust based on your GPU setup |
|
|
) |
|
|
# Generate response |
|
|
sampling_params = SamplingParams(temperature=0.7, max_tokens=512) |
|
|
response = model.generate("Describe this image: <image>", sampling_params) |
|
|
print(response[0].outputs[0].text) |
|
|
``` |
|
|
|
|
|
## 🚀 Key Features |
|
|
- **Vision-Language Optimized**: Specialized quantization recipe that preserves visual understanding |
|
|
- **vLLM Ready**: Seamless integration with vLLM for production deployment |
|
|
- **Memory Efficient**: ~50% memory reduction compared to FP16 original |
|
|
- **Performance Boost**: Significant faster inference on H100/L40S GPUs |
|
|
## 📊 Model Details |
|
|
- **Original Model**: [OpenGVLab/InternVL3-8B](https://huggingface.co/OpenGVLab/InternVL3-8B) |
|
|
- **Source Model**: OpenGVLab/InternVL3-8B |
|
|
- **Quantized Model**: InternVL3-8B-FP8-Dynamic |
|
|
- **Quantization Method**: FP8 Dynamic (W8A8) |
|
|
- **Quantization Library**: [LLM Compressor](https://github.com/vllm-project/llm-compressor) v0.5.2.dev112+g6800f811 |
|
|
- **Quantized by**: [brandonbeiler](https://huggingface.co/brandonbeiler) |
|
|
|
|
|
## 🏗️ Technical Specifications |
|
|
### Hardware Requirements |
|
|
- **Inference**: 7.8GB VRAM (+ Context) |
|
|
- **Supported GPUs**: H100, L40S, A100 (80GB), RTX 4090 (2x for tensor parallelism) |
|
|
- **GPU Architecture**: Ada Lovelace, Hopper (for optimal FP8 performance) |
|
|
### Quantization Details |
|
|
- **Weights**: FP8 E4M3 with dynamic per-tensor scales |
|
|
- **Activations**: FP8 E4M3 with dynamic per-tensor scales |
|
|
- **Preserved Components**: Vision tower, embeddings, normalization layers, mlp1 |
|
|
## 🔬 Package Versions |
|
|
This model was created using: |
|
|
``` |
|
|
llmcompressor==0.5.2.dev112+g6800f811 |
|
|
compressed-tensors==latest |
|
|
transformers==4.52.4 |
|
|
torch==2.7.0 |
|
|
vllm==0.9.1 |
|
|
``` |
|
|
|
|
|
*Quantized with ❤️ using LLM Compressor for the open-source community* |
