Extend eos_token_id

#35
by Wovchena - opened

Extend eos_token_id with pad token because that's the token stored in processor.tokenizer.eos_token_id. That also simplifies the sample from the model card from

generate_ids = model.generate(**inputs, 
  eos_token_id=processor.tokenizer.eos_token_id, 
  **generation_args
)

to

generate_ids = model.generate(**inputs, 
  **generation_args
)
Wovchena changed pull request title from Update generation_config.json to Extend eos_token_id
Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment