Error with transformers 4.51.3: RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

#34

by david-crynge - opened 3 days ago

3 days ago

Using pixtral-12b with transformers-4.51.3 gives the following error:
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

Downgrading to transformers-4.48.3 fixes it, but I'd like to be able to use the latest transformers version

RaushanTurganbay

Unofficial Mistral Community org 2 days ago

@david-crynge can you share your inference code? Seems like you need to cast inputs to correct dtype with inputs = inputs.to(torch.float16) before doing generation

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment