Introducing GGUF-my-LoRA
•
13
Xuan Son NGUYEN
ngxson
AI & ML interests
Doing AI for fun, not for profit
Recent Activity
upvoted
a
collection
about 18 hours ago
Jan 17 Releases ❄️
updated
a collection
3 days ago
GGUF LoRA adapters
updated
a model
3 days ago
ggml-org/LoRA-Deepthink-Reasoning-Qwen2.5-7B-Instruct-Q8_0-GGUF
Articles
Organizations
ngxson's activity
upvoted
a
collection
about 18 hours ago
updated
a
collection
3 days ago
updated
a
model
3 days ago
published
a
model
3 days ago
replied to
their
post
3 days ago
Yes, sure!
The first step is to generate the PEFT-compatible LoRA adapter, I used mergekit-extract-lora to do that. Please note that some bigger models (Qwen/Llama 70B) give some errors that I don't know how to fix, hopefully they will fix that soon. You can find more info about mergekit here: https://github.com/arcee-ai/mergekit
Next step is to convert PEFT to GGUF, I used this space: https://huggingface.co/spaces/ggml-org/gguf-my-lora
Then it's good to go!
Please note that, the space can convert any PEFT LoRA adapters to GGUF, so if you're using something like unsloth, it will be straight-forward to convert into GGUF LoRA (so no need to merge to base model)
upvoted
a
collection
3 days ago
liked
a
model
3 days ago
upvoted
an
article
3 days ago
Article
Run ComfyUI workflows for free on Spaces
•
34
Deployment as server?
10
#1 opened 4 days ago
by
ngxson
upvoted
a
collection
3 days ago
replied to
their
post
4 days ago
Tagging @bartowski @MaziyarPanahi and @mradermacher , you may want to give this a try!
posted
an
update
4 days ago
Post
1571
Check out my collection of pre-made GGUF LoRA adapters!
This allow you to use both normal + abliterated version of popular models like llama, qwen, etc, without having to double to amount of VRAM usage.
ngxson/gguf_lora_collection
This allow you to use both normal + abliterated version of popular models like llama, qwen, etc, without having to double to amount of VRAM usage.
ngxson/gguf_lora_collection
-
- 4 replies
liked
a
model
6 days ago
updated
a
Space
6 days ago
updated
a
model
6 days ago
reacted to
bartowski's
post with 👀👍
7 days ago
Post
18260
Looks like Q4_0_N_M file types are going away
Before you panic, there's a new "preferred" method which is online (I prefer the term on-the-fly) repacking, so if you download Q4_0 and your setup can benefit from repacking the weights into interleaved rows (what Q4_0_4_4 was doing), it will do that automatically and give you similar performance (minor losses I think due to using intrinsics instead of assembly, but intrinsics are more maintainable)
You can see the reference PR here:
https://github.com/ggerganov/llama.cpp/pull/10446
So if you update your llama.cpp past that point, you won't be able to run Q4_0_4_4 (unless they add backwards compatibility back), but Q4_0 should be the same speeds (though it may currently be bugged on some platforms)
As such, I'll stop making those newer model formats soon, probably end of this week unless something changes, but you should be safe to download and Q4_0 quants and use those !
Also IQ4_NL supports repacking though not in as many shapes yet, but should get a respectable speed up on ARM chips, PR for that can be found here: https://github.com/ggerganov/llama.cpp/pull/10541
Remember, these are not meant for Apple silicon since those use the GPU and don't benefit from the repacking of weights
Before you panic, there's a new "preferred" method which is online (I prefer the term on-the-fly) repacking, so if you download Q4_0 and your setup can benefit from repacking the weights into interleaved rows (what Q4_0_4_4 was doing), it will do that automatically and give you similar performance (minor losses I think due to using intrinsics instead of assembly, but intrinsics are more maintainable)
You can see the reference PR here:
https://github.com/ggerganov/llama.cpp/pull/10446
So if you update your llama.cpp past that point, you won't be able to run Q4_0_4_4 (unless they add backwards compatibility back), but Q4_0 should be the same speeds (though it may currently be bugged on some platforms)
As such, I'll stop making those newer model formats soon, probably end of this week unless something changes, but you should be safe to download and Q4_0 quants and use those !
Also IQ4_NL supports repacking though not in as many shapes yet, but should get a respectable speed up on ARM chips, PR for that can be found here: https://github.com/ggerganov/llama.cpp/pull/10541
Remember, these are not meant for Apple silicon since those use the GPU and don't benefit from the repacking of weights
- 16 replies
updated
a
dataset
7 days ago
🚩 Report: Not working
3
#1 opened 8 days ago
by
khang119966
updated
a
collection
7 days ago