Device set to dtype
#2
by cdreetz - opened
in ultravox_model.py there is projector.to(config.torch_dtype)
i dont think this is right?
Module.to is overloaded and can take as input a device or a dtype (or in fact a memory_format or a tensor to copy info from): https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.to
Does that answer your question or did I misunderstand?
pipe = transformers.pipeline(
model='fixie-ai/ultravox-v0_5-llama-3_2-1b',
torch_dtype=torch.float32, # Explicit dtype specification
device=0 if torch.cuda.is_available() else -1, # 0 = first GPU
trust_remote_code=True
) don't forget import torch