electroglyph/gemma-3-4b-it-unslop-GRPO-v2

Gemma 3 4b unslop experiment v2

An unslop finetune of google/gemma-3-4b-it

Updates / Observations

An updated version of this model is here: v3

---

I've received some excellent feedback.

Some usage notes: Low temp recommended. My training technique uses high temp to try to hit slop edge cases, but I ended up baking in some trippiness on accident I think.

Overall I'm starting to like this model. I'm going to adjust things a little bit for my next attempt and bring it back down to earth a bit, but overall it's still creative and less AI-like in a lot of ways.

Changes from my first test

• I created a lot more varied example text from which I grabbed overused bigrams and trigrams. It's now 60MB of junk...I'm starting to dream about em dashes in the rain.
• I completely re-did my datasets with lots of different prompt styles
• Slop examples now number around 6000 in my training script. Lots of bigrams are duplicated in the trigrams, that's mostly a feature and not a bug
• My 4 comma regex rule is activated about 80% thru training. First time around it was active the whole time and made the model output much shorter sentences. I'm trying to achieve a better balance this time
• Trained on about double the amount of tokens
• Model is still a bit underfit. I feel like I'm approaching the brain damage zone so I'm being cautious
• I've uploaded a UD-Q4_K_XL GGUF with settings that I grabbed from Unsloth's quant using my lil utility: quant_clone

Training technique:

I have a pretty complex reward system, so parts are activated in 3 separate stages.

I generated lots of sample text and then sorted all bigrams and trigrams by frequency.

I added some of these to the reward function and penalized their use.

I also added some regex filters for various things.

If the prompt doesn't include "rain", but model output includes rain, it gets penalized.

Same thing for "air". Gemma 3 LOVES to talk about rain and how the air tastes (or clings, etc.)... no more.

Many of my training prompts include a word count for the model to output. Low deviation is rewarded, the opposite is penalized.

Halfway through training I activated lexical diversity comparison. It penalizes MTLD < 100, gives increasing rewards up to 120.

About 80% through training I enabled the 4+ comma per sentence regex which penalizes complex phrases.

There's a callback for early stopping if reward stays high, but it didn't kick in this run.

This was trained on ~30 million tokens on a single 3090. I'm sharing my code so people can try their own finetuning runs.

training code: train.py

Note: some of the bigrams/trigrams look funny because I've stripped any non-alpha chars from them. If you wanna use em you'll have to process your text the same way I did.