pszemraj

/

long-t5-tglobal-base-16384-book-summary

like 135

If I keep your Python instructions but replace your model with another, I get reasonable output:

import torch
from transformers import pipeline

summarizer = pipeline(
    "summarization",
    #"pszemraj/long-t5-tglobal-base-16384-book-summary",                                                                                                                                                    
    "stevhliu/my_awesome_billsum_model",
    device=0 if torch.cuda.is_available() else -1,
)
long_text = "Here is a lot of text I don't want to read. Replace me"

result = summarizer(long_text)
print(result[0]["summary_text"])

Gives:

Here is a lot of text I don't want to read. Replace me with a text that I'm not sure I'd like to use.

It's not a great summary, but the fact that it "works" indicates to me that there might be an issue with pszemraj/long-t5-tglobal-base-16384-book-summary. Is there a way I can validate that the model I downloaded isn't somehow corrupt?

Thanks again,

Matthew

hi! so the output text did not use to be like that at the time of creation of this model. I think there is an issue with the Long-T5 modeling code in transformers in general. I would recommend trying with the following package versions:

pip install -q transformers==4.31.0
pip install accelerate==0.20.3 -q
pip install sentencepiece  -q

This what I have set for the colab for my XL model based on this arch. I haven't had time to investigate this/raise an issue in the transformers repo, but rolling back to earlier versions should help

edit:

Yep, that seems to be the case. check out this google colab notebook where I installed those versions, the output is reasonable with "This is a very long list of text I do not want to read, replace me with you. Replace me."

Thanks for your very quick response. I was pleased to see that your XL model with the latest version of transformers performs as expected.

Thanks again,

Matthew

happy to help and thanks for flagging this issue! I'll add some notes to the model card(s)

Hello,

I'm reopening this discussion to provide a fix with the latest huggingface transformers library. The gibberish output issue is resolved by using the pytorch_model.bin model instead of the safetensors version. A similar issue was reported on Stack Overflow: (https://stackoverflow.com/questions/79180415/huggingface-model-loaded-from-the-disk-generates-gibberish).
The fix is to pass the argument use_safetensors=False :

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("pszemraj/long-t5-tglobal-base-16384-book-summary")
model = AutoModelForSeq2SeqLM.from_pretrained(
  "pszemraj/long-t5-tglobal-base-16384-book-summary",
    use_safetensors=False  # <--- disables safetensors, uses pytorch_model.bin
)

conversation = '''Navyseal copypasta'''
inputs = tokenizer(conversation, return_tensors="pt", max_length=16384, truncation=True)

output_ids = model.generate(
    inputs.input_ids,
    max_length=1024,
    use_cache=True
)

summary = tokenizer.decode(output_ids[0], skip_special_tokens=True)

print(f"Summary: {summary}")
# Summary: The narrator tells us that he's been training in the Navy seals and has been involved in many secret raids. He's got over 300 confirmed deaths, and he can kill you anywhere in the world. You're just another target for him. He promises to wipe out your misery with his "precision" weapons.

Interesting find - let me create an issue on the transformers repo, hopefully it will be easier to pin down and solve with this info!

Apologies for not getting to this sooner, with so many things in flux in ai/software solutions it sometimes feels hopeless on where to start 🙃

I created this issue on the transformers github thanks to the info you provided. Hopefully we can get this resolved!

Thanks for creating the issue :) The transformers pipeline doesn't work for this model as we can't pass the use_safetensors argument. Fortunately, it's still possible to export the pytorch model to ONNX so performance/pickling isn't really an issue.

PR was merged! next release (or install from source) and hopefully these issues are gone forever (🤞 )

https://github.com/huggingface/transformers/pull/40724

Glad to see it's been fixed. I'm looking forward to testing it in the next release 😄