mlabonne

/

gemma-3-27b-it-abliterated

like 165

Would love to see a QAT version of Gemma 3 as well.

same, hope it makes it

please make the qat version

QAT-q4_0-gguf would be perfect for this

Please, sir :)

Hey good news, I abliterated all the QAT unquantized models:

https://huggingface.co/mlabonne/gemma-3-1b-it-qat-abliterated
https://huggingface.co/mlabonne/gemma-3-4b-it-qat-abliterated
https://huggingface.co/mlabonne/gemma-3-12b-it-qat-abliterated
https://huggingface.co/mlabonne/gemma-3-27b-it-qat-abliterated

They need to be quantized but the checkpoints are available.

Thanks for QATs.

I tested your newest 27b. Unfortunately v2 is lobotomized and broken compared to v1. when given the same prompt, it either:

gemma-3-27b-it-abliterated-v2.q4_k_m - doesn't follow instruction and quickly devolves into endless repetition loops.

gemma-3-27b-it-qat-abliterated.q5_k_m - follows instructions partially but less than v1, and with repetitive, broken grammar. also hallucinates context, goes off-topic.

v1 (gemma-3-27b-it-abliterated.q4_k_m) gives thorough, accurate and detailed replies in comparison.

Examples of bugged output:

Okay, here are some ways to explain the above topic. There are many aspects to consider and many different approaches to take.

Here' that topic broken down:

* **The Scope of the Problem** 
* **Here’s a breakdown of those aspects:**

Here's a breakdown of the topic:

* **The Basics**
* *Definition:* A term used in medicine to describe the process of development.
* Here's a breakdown of the above topics:

* **The Basics**
* **Here's a breakdown of the above topics as well as some more details:

* **The Basics**
* **Here's a breakdown of the above topics as well as some others:**
* And finally, here's a breakdown of the above topics:

* **The Basics**
* **Here's a breakdown of the above topics as well as some others:
* Here's a breakdown of the above topics:

These descriptions aim to capture the essence of war through language, while avoiding words like 'suffering', 'humanity' and 'compassion'. Here are some more examples:

      The devastation and destruction: War is often described in terms of its physical impact – the broken buildings, scorched earth, and loss of life.
      The chaos and disorder: War disrupts lives, economies, and societies.
      The cost and expense: War requires resources, manpower, and time.


Here are a few more ways to describe war, keeping in mind the original request:

      The outcome of war: The end result of war is often death and destruction.
      The effects of war: War has many consequences, including economic hardship and social upheaval.
      The purpose of war: Some believe that war is necessary for progress, while others see it as an inevitable part of human nature.

Thanks for flagging this @TheDevilishOne , it should be fixed now. It's something I couldn't see because the outputs I check are truncated. Hopefully, this new version behaves better.

Thanks @mlabonne , I'll test some more but my initial run of the newest 27B v2 Q4_K_M appears to still have bugs.

Would it be possible to apply your v1 abliteration method to the Gemma3 27B QAT? That or create a QAT of this v1 27B?

I found neither the 4B or 12B functioned well for me. But your Gemma3 27B v1 (non-QAT) works pretty much flawlessly. This is my favorite model currently.

Good work sir, you are paving new ground for sure.

Ah, interesting. Are you talking about the 4B and 12B v1 or v2?

After further testing, it appears the v2 for the smaller models works better than their v1 counterparts, but still have some issues.

Whereas with 27B, v1 is the clear winner and the best of the batch. Tested these with a few different prompts using the same settings as the 27B:

gemma-3-4b-it-abliterated.q8_0,https://huggingface.co/mlabonne/gemma-3-4b-it-abliterated-GGUF
This one (v1 of 4b) is completely censored.

gemma-3-4b-it-abliterated-v2.q8_0,https://huggingface.co/mlabonne/gemma-3-4b-it-abliterated-v2-GGUF
This one isn't censored but is somewhat repetitive and lobotomized.

gemma-3-4b-it-qat-abliterated.q8_0.gguf,https://huggingface.co/mlabonne/gemma-3-4b-it-qat-abliterated-GGUF
This one is partially censored and has even more repetition and nonsense output.

gemma-3-12b-it-abliterated.q4_k_m,https://huggingface.co/mlabonne/gemma-3-12b-it-abliterated-GGUF
This one (v1 of 12b) is also completely censored. It likes to insult the user for asking questions that it doesn't like.

gemma-3-12b-it-abliterated-v2.q4_k_m,https://huggingface.co/mlabonne/gemma-3-12b-it-abliterated-v2-GGUF
Just like the v2 non-QAT 4B, this one isn't censored, but has repetition and lobotimize bugs (somewhat nonsensical output).

gemma-3-12b-it-qat-abliterated.q4_k_m,https://huggingface.co/mlabonne/gemma-3-12b-it-qat-abliterated-GGUF
This one isn't censored but varies wildly between giving no response, giving a repetitive one, and giving a high quality response.

gemma-3-27b-it-abliterated.q4_k_m, https://huggingface.co/mlabonne/gemma-3-27b-it-abliterated-GGUF
As tested before, v1 of the 27B is uncensored and has none of the bugs associated with 4B or 12B. It's near perfect.

My conclusion is that v2 of the 4B and 12B are partially usable and an improvement to v1, but unstable compared to the 27B v1.

Thanks again for all your work @mlabonne

Thanks a lot for this comparison @TheDevilishOne !