Mistral-Nemo-Instruct-2407 compatibility?
#1
by
JermemyHaschal
- opened
Hello,
I wanted to know, is this draft model compatible with Mistral-Nemo-Instruct-2407? And if not, could you perhaps make one?
The vocab seems to be the same so it should work, although the performance improvement may not be as good. Try it and let me know how it performs!
I tried your draft model with Mistral Nemo 12B (specifically, the Rocinante finetune), and it went from 41T/s to 31T/s on my computer. So that's a 25% performance decrease, unfortunately.
Oh, that's expected for a highly tuned model like Rocinante, I looks like it have a distinct style far from "standard" chat models