Embedding Model uses unsloth library instead of transformers
Hello, i have RAG system.
Having the LLM loaded by unsloth, it seems there is conflict when unsloth library is loaded. I am using sentence transformers to load the qwen3 0.6B embedding model.
The problem didn't appear when i created the vector database because there was no load for the unsloth library.
Traceback:
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards: 33%|███▎ | 1/3 [00:01<00:03, 1.58s/it]
Loading checkpoint shards: 67%|██████▋ | 2/3 [00:03<00:01, 1.49s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:03<00:00, 1.01s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:03<00:00, 1.15s/it]
Batches: 0%| | 0/1 [00:00<?, ?it/s]
Batches: 0%| | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/mahmoud.hassanain/phi-inference/memo_rg/qwen-emb/memo_rg_2_qwen3_embedder.py", line 720, in
emb = model_encoder.encode([original_query], prompt_name="reason", show_progress_bar=True, device='cuda', normalize_embeddings=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mahmoud.hassanain/.local/lib/python3.11/site-packages/sentence_transformers/SentenceTransformer.py", line 685, in encode
out_features = self.forward(features, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mahmoud.hassanain/.local/lib/python3.11/site-packages/sentence_transformers/SentenceTransformer.py", line 758, in forward
input = module(input, **module_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mahmoud.hassanain/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mahmoud.hassanain/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mahmoud.hassanain/.local/lib/python3.11/site-packages/sentence_transformers/models/Transformer.py", line 442, in forward
outputs = self.auto_model(**trans_features, **kwargs, return_dict=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mahmoud.hassanain/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mahmoud.hassanain/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mahmoud.hassanain/.local/lib/python3.11/site-packages/unsloth/models/llama.py", line 862, in LlamaModel_fast_forward
position_embeddings = self.rotary_emb(hidden_states, position_ids, self.config.max_position_embeddings)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mahmoud.hassanain/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mahmoud.hassanain/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mahmoud.hassanain/.local/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
TypeError: Qwen3RotaryEmbedding.forward() takes 3 positional arguments but 4 were given
I guess it's an unsloth issue, and not the Qwen3 model itself, huh?
it could be from unsloth but still something doesn't make sense that the problem happens only when loading and using qwen3 embedding meanwhile other embedding models working fine