Error when running demo 1 to 1
Hi, I am trying to run the model locally, with the exact same setup as the demo. I am on a Mac M4 Max 64GB. Tested with Molmo-7B-D-0924 and Molmo-72B-0924 (with offloading).
It loads fine but when I try to run generate_from_batch, I get the following error:
outputs = self.model.forward(
File "~/.cache/huggingface/modules/transformers_modules/ac032b93b84a7f10c9578ec59f9f20ee9a8990a2/modeling_molmo.py", line 1836, in forward
past_length = past_key_values[0][0].size(-2)
AttributeError: 'NoneType' object has no attribute 'size'
I've tried searching for a solution but couldn't find anything. Does anyone have an idea why the demo setup would be crashing like that?
Edit: I also tried wrapping with with torch.autocast(device_type="mps", enabled=True, dtype=torch.bfloat16): and running model.to("mps") before that but it's the same (cpu or mps).
Edit2: SOLVED. Had to downgrade transformers to 4.50.3, which is the last version it's patched for, above 4.51 the cache object behaves differently.
Any solution to this?
Any solution to this?
Yes, I added the solution as an edit on the last line. It was caused by the transformers version, it should be 4.50.3, not greater.