Llama-3.1 Quantization - a neuralmagic Collection

neuralmagic 's Collections

Sparse-Llama-3.1-2of4

Vision Language Models Quantization

FP8 LLMs for vLLM

Llama-3.2 Quantization

Llama-3.1 Quantization

INT8 LLMs for vLLM

INT4 LLMs for vLLM

Sparse Foundational Llama 2 Models

Compression Papers

DeepSparse Sparse LLMs

Sparse Finetuning MPT

Compressed LLMs from the Community

Llama-3.1 Quantization

updated Nov 22, 2024

Neural Magic quantized Llama-3.1 models

neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8

Text Generation • Updated Oct 9, 2024 • 37.4k • 37
neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8

Text Generation • Updated Oct 9, 2024 • 329k • 37
neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w4a16

Text Generation • Updated Oct 10, 2024 • 7.32k • 28
neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w4a16

Text Generation • Updated Dec 17, 2024 • 6.77k • 23
neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w8a16

Text Generation • Updated Oct 23, 2024 • 12.6k • 9
neuralmagic/Meta-Llama-3.1-405B-Instruct-quantized.w4a16

Text Generation • Updated Oct 10, 2024 • 295 • 12
neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8-dynamic

Text Generation • Updated Oct 19, 2024 • 214 • 14
neuralmagic/Meta-Llama-3.1-405B-Instruct-quantized.w8a8

Text Generation • Updated Dec 3, 2024 • 366 • 2
neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8-dynamic

Text Generation • Updated Oct 19, 2024 • 1.11k • 5
neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w8a8

Text Generation • Updated Oct 10, 2024 • 7.43k • 19
neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic

Text Generation • Updated Oct 19, 2024 • 2.32k • 5
neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8

Text Generation • Updated Oct 9, 2024 • 1.31k • 31
neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w8a8

Text Generation • Updated Oct 23, 2024 • 3.3k • 13
neuralmagic/Meta-Llama-3.1-70B-Instruct-quantized.w8a16

Text Generation • Updated Oct 9, 2024 • 222 • 4
neuralmagic/Meta-Llama-3.1-8B-FP8

Text Generation • Updated Oct 9, 2024 • 2.47k • 5
neuralmagic/Meta-Llama-3.1-70B-FP8

Text Generation • Updated Oct 9, 2024 • 330 • 1
neuralmagic/Meta-Llama-3.1-8B-quantized.w8a16

Text Generation • Updated Oct 9, 2024 • 14.2k • 1
neuralmagic/Meta-Llama-3.1-8B-quantized.w8a8

Text Generation • Updated Oct 23, 2024 • 204 • 2
neuralmagic/Meta-Llama-3.1-405B-FP8

Text Generation • Updated Oct 9, 2024 • 57
neuralmagic/Meta-Llama-3.1-405B-Instruct-quantized.w8a16

Text Generation • Updated Oct 9, 2024 • 139 • 2
neuralmagic/Meta-Llama-3.1-70B-quantized.w8a8

Text Generation • Updated Oct 9, 2024 • 7