google

/

gemma-3-4b-it

like 736

Follow

Google 21.8k

Hello,

I am struggling to get this model to work. The provided example code does not produce any sort of coherent output in my case.
All I did, was copy the example code, paste it into my IDE and start it.

It runs... but the output is:

WARNING:accelerate.big_modeling:Some parameters are on the meta device because they were offloaded to the cpu.
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
The following generation flags are not valid and may be ignored: ['top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
Here's what you're using** based on the information provided**** in the image, **are you still able to provide****
**...**

******

******
****
**are**** ******

******
****
****
****
****
****
****
****

****
****
****
****
****
****
****
**

This is the code I used (copied verbatim from the README.md):

# pip install accelerate

from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from PIL import Image
import requests
import torch

model_id = "google/gemma-3-4b-it"

model = Gemma3ForConditionalGeneration.from_pretrained(
    model_id, device_map="auto"
).eval()

processor = AutoProcessor.from_pretrained(model_id)

messages = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "You are a helpful assistant."}]
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"},
            {"type": "text", "text": "Describe this image in detail."}
        ]
    }
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device, dtype=torch.bfloat16)

input_len = inputs["input_ids"].shape[-1]

with torch.inference_mode():
    generation = model.generate(**inputs, max_new_tokens=100, do_sample=False)
    generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)

This is the libraries I have installed:

accelerate==1.9.0
black==24.4.2
certifi==2025.7.14
charset-normalizer==3.4.2
click==8.2.1
coverage==7.6.12
debugpy==1.8.15
filelock==3.18.0
fsspec==2025.7.0
hf-xet==1.1.5
huggingface-hub==0.33.4
idna==3.10
importlib_metadata==8.7.0
iniconfig==2.1.0
JayDeBeApi==1.2.3
Jinja2==3.1.6
jpype1==1.6.0
MarkupSafe==3.0.2
mpmath==1.3.0
mypy_extensions==1.1.0
networkx==3.5
numpy==2.3.1
nvidia-cublas-cu12==12.6.4.1
nvidia-cuda-cupti-cu12==12.6.80
nvidia-cuda-nvrtc-cu12==12.6.77
nvidia-cuda-runtime-cu12==12.6.77
nvidia-cudnn-cu12==9.5.1.17
nvidia-cufft-cu12==11.3.0.4
nvidia-cufile-cu12==1.11.1.6
nvidia-curand-cu12==10.3.7.77
nvidia-cusolver-cu12==11.7.1.2
nvidia-cusparse-cu12==12.5.4.2
nvidia-cusparselt-cu12==0.6.3
nvidia-nccl-cu12==2.26.2
nvidia-nvjitlink-cu12==12.6.85
nvidia-nvtx-cu12==12.6.77
packaging==25.0
pathspec==0.12.1
pillow==11.3.0
platformdirs==4.3.8
pluggy==1.6.0
psutil==7.0.0
pytest==8.2.2
pytest-cov==5.0.0
python-dotenv==1.1.1
PyYAML==6.0.2
regex==2024.11.6
requests==2.32.4
safetensors==0.5.3
setuptools==80.9.0
sympy==1.14.0
tabulate==0.9.0
tokenizers==0.21.2
torch==2.7.1
torchao==0.12.0
tqdm==4.67.1
transformers==4.53.3
triton==3.3.1
typing_extensions==4.14.1
urllib3==2.5.0
zipp==3.23.0

Does anybody have an idea as to why this is happening?
Thank you kindly.