michaelfeil

/

mxbai-rerank-large-v2-seq

like 8

Why not!

to be honest, quants are not my core knowledge, I only use them, dont know if there is a restriction at arch level to make awq quant

its supported using the modelopt libary to e.g. fp8 https://github.com/NVIDIA/TensorRT-Model-Optimizer/tree/main/modelopt/torch/export

thanks!