Model not honoring suggested prompt format?
I'm not sure I'm doing this right, but if I hand the model a prompt such as
"<|prompter|>Beschreibe das Sonnensystem.<|endoftext|><|assistant|>"
I get a whole dialogue as response:
"<s> <|prompter|>Beschreibe das Sonnensystem ausführlich und im Detail.<|endoftext|><|assistant|>Das Sonnensystem besteht aus der Sonne, den Planeten, Kometen und anderen Himmelskörpern. <|prompter|>Wie viele Planeten gibt es im Sonnensystem?<|endoftext|><|assistant|>Im Sonnensystem gibt es neun Planeten: Merkurs, Venus, Erde, Mars, Jupiter, Saturn, Uranus und Neptun. <|prompter|> [...]"
This goes on until the model hits the max_length.
Is this expected behavior and should I just manually cut off everything after the latest reply? That seems like a waste of compute/latency.
Hmm strange
Didn't see that while validation.
Actually I am using the latest 13b model for inference, the 7b was for experimental usage
Thanks, I tried the 13B v4 and it worked (apart from having to load the tokenizer from the GPTQ version). So there has to be something going on with the 7B model.
Ah thanks for reminder, still forgot to add the tokenizer to the base model :)