Repetition issue
#1
by
lazyDataScientist
- opened
Could you share the exact parameters that you use which the model returns good output. I have tried the recommended repetition_penalty setting and I get !?!?!?!?!?!?!?!?!?!... for every prompt that that I gave the model.
Sorry, never got a notification for this comment for some reason.
That type of repetition looks like a different problem. Could be prompt format (llama-chat) or rope scaling (which has to be 8)?
Here are 2 sets that work well for me in Oobabooga:
Standard sampling:
'temperature': 0.8
'top_p': 0.6
'min_p': 0
'top_k': 40
'repetition_penalty': 1.12
'presence_penalty': 0
'frequency_penalty': 0
'repetition_penalty_range': 1024
'typical_p': 1
'tfs': 1
'top_a': 0
Mirostat:
'mirostat_mode': 2
'mirostat_tau': 2
'mirostat_eta': 0.1
with the other settings set to defaults:
'temperature': 1
'top_p': 1
'min_p': 0
'top_k': 0
'repetition_penalty': 1
'presence_penalty': 0
'frequency_penalty': 0
'repetition_penalty_range': 1024
'typical_p': 1
'tfs': 1
'top_a': 0
edit: I see we actually sorted this out in a different thread, didn’t connect the names… Well these settings can be here for others.