sandman4/Qwen3-32B-GPTQ-8bit-float16

Qwen3-32B Quantized Model

8-bit quantized with float16 version of Qwen3-32B using gptqmodel.

Quantization

from datasets import load_dataset
from gptqmodel import GPTQModel, QuantizeConfig
import sys
import torch

model_id = sys.argv[1]
quant_path = "quantized_model"

# Load calibration data (1024 samples from C4)
calibration_dataset = load_dataset(
    "allenai/c4",
    data_files="en/c4-train.00002-of-01024.json.gz",
    split="train"
  ).select(range(1024))["text"]

# Configure and run quantization
quant_config = QuantizeConfig(bits=8, group_size=128)
model = GPTQModel.load(model_id, quant_config, torch_dtype=torch.float16)
model.quantize(calibration_dataset, batch_size=2)
model.save(quant_path)

License

Apache-v2. See LICENSE.txt