unsloth

/

gemma-3-12b-it-GGUF

like 66

Follow

Unsloth AI 5.31k

That was Q3_K_XL I think. I deleted it and tried IQ2_XXS, and even though it comfortably fits in VRAM (RTX3080 10GB), it fails right at the point that it seems fully loaded.

Debug log analysis according to o4-mini:
The culprit here is buried in these two adjacent blocks:

clip_ctx: CLIP using CUDA0 backend
clip_model_loader: model name:   Gemma-3-12B-It-Qat
…
load_hparams: vision_encoder:     1
…
load_hparams: model size:         134217949.38 MiB
…
llama.cpp abort:2743: GGML_ASSERT(ggml_can_mul_mat(a, b)) failed

What’s happening is that LM Studio is trying to spin up a CLIP-style vision encoder in addition to your text LLM, and—by misconfiguration—it’s pointed at your Gemma-3-12B-It-Qat .gguf file instead of an actual CLIP model. You can see that the loader thinks your Gemma model has a “vision_encoder” component and reports an absurd size (~128 TB!), which obviously isn’t right. When llama.cpp then tries to do a matrix multiply with tensors whose dimensions don’t line up, it trips the ggml_can_mul_mat(a, b) assertion and aborts.

How to fix

1. Disable the vision encoder if you don’t need any image-to-text capability.
   In your LM Studio project settings, turn off or remove the CLIP/Vision encoder entry so that only the LLM is loaded.

2. Point to a real CLIP model if you do intend to use a multimodal (vision+LLM) setup.
   Download an appropriate GGUF CLIP ViT file (e.g. clip-vit-base-patch32.gguf) and in your project’s “Vision Encoder” slot, reference that file instead of the Gemma LLM.

Once the CLIP loader is either disabled or correctly configured, the mismatched-shapes error will go away and your Gemma model will load normally.

Same with 4b model. Is it just me??

Apparently according to some people, you'll need to wait for the latest LM Studio update. Do you know if it runs fine in llama.cpp?

It fails when using both F16 and F32 mmproj with the latest llama-mtmd-cli in llama.cpp releases(b5216). It works when switching to https://huggingface.co/lmstudio-community/gemma-3-12b-it-GGUF/blob/main/mmproj-model-f16.gguf.

How to fix

1. Disable the vision encoder if you don’t need any image-to-text capability.
   In your LM Studio project settings, turn off or remove the CLIP/Vision encoder entry so that only the LLM is loaded.

2. Point to a real CLIP model if you do intend to use a multimodal (vision+LLM) setup.
   Download an appropriate GGUF CLIP ViT file (e.g. clip-vit-base-patch32.gguf) and in your project’s “Vision Encoder” slot, reference that file instead of the Gemma LLM.

Once the CLIP loader is either disabled or correctly configured, the mismatched-shapes error will go away and your Gemma model will load normally.

I cant figure out how to disable the vision part in LM Studio. If I am use chat\server capabilities in the program.

@danihend Please do not post "fixes" you get from LLMs without verifying them first.

@danihend Please do not post "fixes" you get from LLMs without verifying them first.

I didn't post a fix, I just shared what o4-mini's analysis was.

That's why I said "Debug log analysis according to o4-mini:"

Anyone browsing this forum is certainly grown up enough to decide if they believe it or not. Did you take some irreversible actions based on it's advice or what?

@danihend Your message contains more text than the debug log, you also wrote "how to fix". This is a hallucination, and thus spam. Stop posting LLM hallucinations here. You meant to be helpful, great, thanks for that. But hallucinations are not helpful.

@danihend Your message contains more text than the debug log, you also wrote "how to fix". This is a hallucination, and thus spam. Stop posting LLM hallucinations here. You meant to be helpful, great, thanks for that. But hallucinations are not helpful.

I didn't write that, o4 did. I am not "posting hallucinations" . Please learn how to interact with people in a normal way or just keep it to yourself.