google

/

gemma-2-2b-it

like 1.04k

Follow

Google 10.6k

Hi @vinayakarsh ,

Getting an error because TorchDynamo is trying to optimize the computation graph, but BitsAndBytes 4-bit quantized layers are not fully supported by TorchDynamo. To solve this error, to resolve this error, please disable TorchDynamo using the following code.

  import torch._dynamo
  torch._dynamo.config.suppress_errors = True
  torch._dynamo.disable()

The code was successfully executed in Google Colab with a T4 GPU runtime. You can check the details in the provided gist file, where I have also listed the library versions used.

Thank you.

thanks for the help... I tried disabling TorchDynamo using the code above and it returns the below error:
"""
Unsupported: call_method UserDefinedObjectVariable(Params4bit) t [] {}

from user code:
File "/usr/local/lib/python3.11/dist-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.11/dist-packages/transformers/models/gemma2/modeling_gemma2.py", line 887, in forward
outputs = self.model(
File "/usr/local/lib/python3.11/dist-packages/transformers/models/gemma2/modeling_gemma2.py", line 667, in forward
layer_outputs = decoder_layer(
File "/usr/local/lib/python3.11/dist-packages/transformers/models/gemma2/modeling_gemma2.py", line 321, in forward
hidden_states, self_attn_weights = self.self_attn(
File "/usr/local/lib/python3.11/dist-packages/transformers/models/gemma2/modeling_gemma2.py", line 216, in forward
query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
File "/usr/local/lib/python3.11/dist-packages/bitsandbytes/nn/modules.py", line 484, in forward
return bnb.matmul_4bit(x, self.weight.t(), bias=bias, quant_state=self.weight.quant_state).to(inp_dtype)

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
"""

I had set these using the below code:
"""
import os
os.environ["TORCH_LOGS"] = "+dynamo"
os.environ["TORCHDYNAMO_VERBOSE"] = "1"
"""

still return the same error...

I face the same problem......So how to solve it.

apt install gcc
write "export CC=/usr/bin/gcc " to .bashrc

Could you please share more details about hardware environment and also, sharing the code you are using would be helpful. This information will enable us to better understand the issue and assist you effectively.

Thank you.

using T4 gpu from google colab...

Hi @vinayakarsh ,

I successfully executed code in as your mentioned environment (T4 GPU) in google colab. For more details, please refer to this gist file.

Thank you.

I am also facing the same issue. With 4-bit quantization, during inference in a GPU based machine, I am getting error

2025-03-14 22:59:12,521 - ERROR - Error generating response: call_method UserDefinedObjectVariable(Params4bit) t [] {}

from user code:
File "/home/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
return func(*args, **kwargs)
File "/home/lib/python3.10/site-packages/transformers/models/gemma2/modeling_gemma2.py", line 887, in forward
outputs = self.model(
File "/home/lib/python3.10/site-packages/transformers/models/gemma2/modeling_gemma2.py", line 667, in forward
layer_outputs = decoder_layer(
File "/home/lib/python3.10/site-packages/transformers/models/gemma2/modeling_gemma2.py", line 321, in forward
hidden_states, self_attn_weights = self.self_attn(
File "/home/lib/python3.10/site-packages/transformers/models/gemma2/modeling_gemma2.py", line 216, in forward
query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
File "/home/lib/python3.10/site-packages/bitsandbytes/nn/modules.py", line 484, in forward
return bnb.matmul_4bit(x, self.weight.t(), bias=bias, quant_state=self.weight.quant_state).to(inp_dtype)

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

FYI this seems to be a bug with transformers 4.49.0. Downgrading to 4.48.0 works for me

Confirming that downgrading to 4.48.0 fixes this.

thank you... it was really helpful...