error in latest transformers 4.56.0, AttributeError: 'DynamicCache' object has no attribute 'get_usable_length'. Did you mean: 'get_seq_length'
I was playing with latest 4.56.0 transformers, got error when running the model:
past_key_values_length = past_key_values.get_usable_length(seq_length)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DynamicCache' object has no attribute 'get_usable_length'. Did you mean: 'get_seq_length'?
It is just a workaround, but this worked for me:
model.config.use_cache = False
It is just a workaround, but this worked for me:
model.config.use_cache = False
Thanks, which version of transformers are you using and how did you load the model exactly?
It is just a workaround, but this worked for me:
model.config.use_cache = False
I encountered this issue as well, and this modification did not work. The version is "transformers = 4.56.2".
However, NV-Embed-v2 can work well with "transformers = 4.46.0". Thus, this problem is caused by the update of Transformers.
@le723z
I tried @GrantL10 solution, but that also didn't work. So, I looked at the sentence_transformer config file for the model and installed all the versions of torch, transformers and sentence_transformers that were mentioned & that worked. However, it's an incredibly poorly optimised model. I can run batch size 10 for Qwen-8B, but this, I can only do 1 or 2,..I have a 80 GB A100 GPU & it's still failing on it. With batch size 1, memory utilization is 50GB,.. that's wild tbh lol
I tried @GrantL10 solution, but that also didn't work. So, I looked at the sentence_transformer config file for the model and installed all the versions of torch, transformers and sentence_transformers that were mentioned & that worked. However, it's an incredibly poorly optimised model. I can run batch size 10 for Qwen-8B, but this, I can only do 1 or 2,..I have a 80 GB A100 GPU & it's still failing on it. With batch size 1, memory utilization is 50GB,.. that's wild tbh lol
I haven't used NV-Embed-v2 in depth, and I just used it for running MMAU-Pro benchmark. Perhaps https://github.com/sonalkum/MMAUPro can provide some reference.