Vs rpcal?
Did you ever find a notable difference between this and the 3.00 you made with rpcal? i don't have the means to properly test myself, but i figured you may have some notes on which of the two is a better pick.
It's funny, but the differences are still very subjective. I was unable to quantify things, but always felt like the RPCAL was better. I just ... can't put my finger on WHY it seems that way. Even counting things like "regens per 50-message session" turned out roughly the same. Which tells me that it may be somewhat about prompting/what you say to it in the chat along the way. But the short answer is, no: I prefer the RPCAL, but I can't for the life of me tell you why it ends up that way. And other quantization methods (Imatrix) use random noise to achieve better results for the same quant? So ... this is still a mystery.
And, at this point, I've been using WizardLM-2-8x22b on OpenRouter ... and I might not be able to put up with a less capable model after this. The experience has been so far above what even this model (my previous go-to) is capable of ... I'm spoiled, now. And buying the iron to run 8x22 is "above my pay grade," as they say. So, I may be stuck paying by the token for better and for worse ...
I heard great things about the 8x22, but I'm strictly local so that's a bummer for me haha. I went ahead and used the rpcal and honestly I'm pretty in love with it. I will admit that it's a little... Lackluster when it comes to erp, pretty boring actually, but out of any 30b I've ever used this one feels really great to use in casual rp and tops them all in terms of experience. It actually gives cards a believeable personally instead of just "being the card" (It's hard to explain), but I'm satisfied. I also noticed that the model is INCREDIBLY sensative to system prompts. I swapped it out with a couple I've found online and got drastically different outputs on the same seed. It's rather interesting as I've only ever worked with Yi models untill now where the prompt didn't have such an impact so that's something to tinker with now.
One thing I notice about the 8x7b base is that it is very sensitive to many different kinds of adjustments. This model, in particular, is pretty good, but I agree it can get a bit "vanilla" with ERP ... unless you have the settings dialed in for the samplers. Dynamic temperature was a game changer for me with 8x7 models, as well, because I didn't have to fiddle with 0.01's worth of min-P all the time to prevent going from vanilla to creative to batshit crazy (or vice versa: Off the rails to boring and repetitive). So, that is a hallmark of this model, to me, is that small changes can yield big difference in result (like system prompts, small sampler tuning differences, etc.). For more consistent ERP, I would recommend BagelMIsteryTour-v2 which was my go-to before Wizard 8x22.