google-t5/t5-small · how to use the trained model to infer ?

Response here.

My sample code:

from transformers import AutoTokenizer, T5ForConditionalGeneration
import torch

device:str = 'cuda' if torch.cuda.is_available() else 'cpu'

tokenizer = AutoTokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small").to(device)

for prompt in ["Hello, How are you?", "My name is Arnaud"]:
    print("Input:", prompt)
    inputTokens = tokenizer("translate English to French: {}".format(prompt), return_tensors="pt").to(device)
    outputs = model.generate(inputTokens['input_ids'], attention_mask=inputTokens['attention_mask'], max_new_tokens=50)
    print("Output:", tokenizer.decode(outputs[0], skip_special_tokens=True))