README.md

---
tags:
- fp8
- fp8-dynamic
- vllm
- llm-compressor
- internvl3.5
- internvl
language:
- multilingual
pipeline_tag: image-text-to-text
inference: false
license: mit
base_model: OpenGVLab/InternVL3_5-38B
base_model_relation: quantized
library_name: vllm
---

# InternVL3.5 38B FP8

This is an FP8 dynamically quantized (W8A8) version of `OpenGVLab/InternVL3_5-38B`optimized for high-performance inference with *vLLM*.

The quantization process uses a specialized recipe that preserves the model's core visual understanding capabilities while reducing the memory footprint by nearly 40%.

## Just Run It (vLLM serve)

You can serve the model using vLLM's OpenAI-compatible API server.

```bash
vllm serve brandonbeiler/InternVL3_5-38B-FP8-Dynamic \
    --quantization compressed-tensors \
    --served-model-name internvl3_5-38b \
    --reasoning-parser qwen3 \
    --trust-remote-code \
    --max-model-len 32768 \
    --tensor-parallel-size 1 # Adjust based on your GPU setup
```
**Notes**
- 32k max context length
- reasoning parser ready to go, requires system prompt to run in thinking mode
- still investigating tool calling


## Key Features

*   **Calibration-Free FP8:** Dynamic W8A8 quantization. Weights are pre-quantized, and activations are quantized on the fly.
*   **Vision-Language Optimized:** The vision tower, embeddings, and the first MLP layer are preserved in full precision to maintain high performance on vision-language tasks.
*   **vLLM Ready:** Designed for seamless integration with vLLM for high-throughput serving.
*   **Memory Efficient:** ~40% memory reduction compared to the original FP16 model.
*   **Performance Boost:** Accelerated inference on FP8-compatible hardware (e.g., NVIDIA H100, L40S).

## Model Details

| Attribute | Value |
| :--- | :--- |
| **Original Model** | [OpenGVLab/InternVL3_5-38B](https://huggingface.co/OpenGVLab/InternVL3_5-38B) |
| **Quantized Model** | `brandonbeiler/InternVL3_5-38B-FP8-Dynamic` |
| **Quantization Method** | FP8 Dynamic (W8A8) |
| **Quantization Library** | [LLM Compressor](https://github.com/vllm-project/llm-compressor) v0.7.1 |
| **Quantized By** | [brandonbeiler](https://huggingface.co/brandonbeiler) |


## Usage with vLLM in Python

The following snippet demonstrates inference using the vLLM library.

```python
from vllm import LLM, SamplingParams

# Load the quantized model
# trust_remote_code is required to load the custom model architecture. [32, 44, 45, 48]
model = LLM(
    model="brandonbeiler/InternVL3_5-38B-FP8-Dynamic",
    trust_remote_code=True,
    max_model_len=32768,          # InternVL 3.5 supports a 32k context length. [19, 41]
    tensor_parallel_size=1,      # Adjust for your hardware setup. [11, 15, 38, 40]
)

# Set sampling parameters
# A temperature of 0.6 is recommended for this model. [39]
sampling_params = SamplingParams(temperature=0.6, max_tokens=512)

# Generate a response
# Note: Replace "<image>" with your image input
prompt = "Describe this image: <image>"
response = model.generate(prompt, sampling_params)

print(response[0].outputs[0].text)
```



## Technical Specifications

### Hardware Requirements

*   **Base VRAM:** ~47GB (for model weights)
*   **Context VRAM:**
    *   \+ ~1.3GB for 10k token context
    *   \+ ~2GB for 32k token context with FP8 KV cache
*   **Recommended GPUs:** NVIDIA H100, L40S
*   **Supported GPUs:** NVIDIA A100 (80GB), 2x RTX 4090 (with tensor parallelism), latest AMD GPUs.
*   **Optimal Performance:** NVIDIA GPUs with Compute Capability >= 9.0 (Hopper, Blackwell).

### Quantization Details

*   **Weights:** FP8 E4M3 with per-tensor scales.
*   **Activations:** Dynamically quantized to FP8 E4M3 with per-tensor scales.
*   **Preserved Modules (Full Precision):** Vision tower, embeddings, and the first MLP layer (mlp1).

## Package Versions

This model was quantized using the following environment:

```
llmcompressor==0.7.1
compressed-tensors==0.10.2
transformers==4.55.0
torch==2.7.1
vllm==0.10.1.1
```

*Quantized with ❤️ using [LLM Compressor](https://github.com/vllm-project/llm-compressor) for the open-source community.*