nvidia

/

Llama-3_3-Nemotron-Super-49B-v1_5

like 132

Follow

NVIDIA 33.4k

Download, serve with -q fp8 with vllm.

Warning
Currently, we load the model at original precision before quantizing down to 8-bits, so you need enough memory to load the whole model.

Does it require 2x H100 to load the model before it is quantized to fp8?

Check out the offically released FP8 version at https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5-FP8