autogptq_cuda_256 error
I'm quite new to this so I apologize if this is a noob error. But when I run this example code as is (or uncommenting the revision= block) in Collab (with the T4), I get the following:
/usr/local/lib/python3.10/dist-packages/auto_gptq/nn_modules/qlinear/qlinear_cuda_old.py in init(self, bits, group_size, infeatures, outfeatures, bias, use_cuda_fp16, kernel_switch_threshold, trainable)
81 self.kernel_switch_threshold = kernel_switch_threshold
82 self.autogptq_cuda_available = _autogptq_cuda_available
---> 83 self.autogptq_cuda = autogptq_cuda_256
84 if infeatures % 256 != 0 or outfeatures % 256 != 0:
85 self.autogptq_cuda = autogptq_cuda_64
NameError: name 'autogptq_cuda_256' is not defined
This is a common problem with AutoGPTQ. But fortunately, just today, support for GPTQ has got a lot easier from Python thanks to it now being directly supported from Transformers.
And the HF team have created an awesome Google Colab notebook that shows you how to do everything you could want, for free on Google Colab: https://colab.research.google.com/drive/1_TIrmuKOFhuRRiTWN94iLKUFu6ZX4ceb?usp=sharing
Wow. Thanks for the up-to-the-minute info!