Somewhat broken but interesting
This model does produce some interesting outputs in terms of creative overall direction but it has massive repetition issues, oftentimes within the same response. Anecdotally, it does seem to be infected with more slop than the latest series of usual Beaver/Drummer models. It's a shame because it does seem smart and creative but the constant repetition issues do make it unusable IMO. I'd love to see you keep iterating on it to see if it can be fixed.
I tested with Mistral, Meth, and ChatML. Temp tested at 0.7-1.3 with minP 0.05 - 0.1. No smoothing.
This was an interesting and challenging experiment, but ultimately proved to be a dead end. MoEs are hard.
@goldendase Thank you for the feedback. I'll look into patching it up one more time. Look out for v1c soon!