How to configure koboldcpp?

by kexul - opened Jun 15, 2023

Jun 15, 2023

Hi @TheBloke , Many thanks for your work! May I ask how should I properly configure the koboldcpp to use this model?

Currenty I use Instruct mode with Start Sequence: Below is an instruction that describes a task. Write a response that appropriately completes the request ### Instruction:, with End Sequence: ### Response:.

Here is the full configuration:

However, the model generate some strange string.

TheBloke

Owner Jun 15, 2023

Hi @kexul

I don't have any experience of KoboldCpp myself so I asked Lost Ruins, the koboldcpp developer. He said:

okay looks like they ran the model without unbanning the EOS token. by default, koboldcpp bans the EOS (end of stream) token for all models, allowing infinite text generation
they'd have to run it with the flag --unbantokens and try again

So please try that!

concedo

Jun 15, 2023

•

edited Jun 15, 2023

Can you try launching KoboldCpp with the --unbantokens flag (or if using the GUI, select Unban Tokens) and then try again?

Edit: Took a closer look and realized that starcoder models use a different EOS ID. This will be handled in KoboldCpp release 1.31

kexul

Jun 16, 2023

Many thanks for your suggestion @TheBloke , @concedo , the --unbantokens flag works very well.

So this is my settings for now:

The memory is used to set the prompt, which makes the setting panel more tidy, according to some suggestion I found online:

Hope this helps!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment