Hugging Face's logo
Alan Tseng's picture
10 7

Alan Tseng

agentlans
Gargaz's profile picture jth01's profile picture Memtimintursun's profile picture
·

AI & ML interests

Small data, boring AI

Recent Activity

replied to Dragunflie-420's post 21 minutes ago
Hello community. My name is nikki and I am looking to form a team for a serious project build platform/design/idea/project's...Ive been creating AI professional personas with custom skill sets and divisions of expertise. I want to create a viable business. Ive been working hard but i admit theres so much i do not have time to learn to do. Its taken me three years to learn enough to be here. I dont have a big set up in fact im cloud and ide space trial enterprise here and there all for space. I suck at execution and thats because I dont know how really. I need help from a person. AI has done all it can without hands. Im blabbering at this point. Have nothing big techy to say other than I build and ideate all day hmu glad to meet some like minded individuals ...seriously! Teach me leave me feeling confident in our collaborations not the need to build security software....poor attemt at hacking humor...im neither a comedian or hacker lol....full stacker yep:)
reacted to eaddario's post with 👍 41 minutes ago
Squeezing Tensor Bits: the quest for smaller LLMs An area of personal interest is finding ways to optimize the inference performance of LLMs when deployed in resource-constrained environments like commodity hardware, desktops, laptops, mobiles, edge devices, etc. The method that I'm using to produce these experimental versions, for example https://huggingface.co/eaddario/DeepSeek-R1-Distill-Llama-8B-GGUF is explained in https://medium.com/@eaddario/squeezing-tensor-bits-the-quest-for-smaller-llms-86b23bd052ca At a high level it involves using a custom version of the llama-quantize tool to selectively quantize different tensors at different levels. On average a 10% or more reduction with little loss of quality is possible. There’re two PRs to merge these changes back into the core project but until then, the modified version will be available on GitHub https://github.com/EAddario/llama.cpp/tree/quantize Would love to hear if you can achieve smaller sizes at higher quality!
replied to eaddario's post 42 minutes ago
Squeezing Tensor Bits: the quest for smaller LLMs An area of personal interest is finding ways to optimize the inference performance of LLMs when deployed in resource-constrained environments like commodity hardware, desktops, laptops, mobiles, edge devices, etc. The method that I'm using to produce these experimental versions, for example https://huggingface.co/eaddario/DeepSeek-R1-Distill-Llama-8B-GGUF is explained in https://medium.com/@eaddario/squeezing-tensor-bits-the-quest-for-smaller-llms-86b23bd052ca At a high level it involves using a custom version of the llama-quantize tool to selectively quantize different tensors at different levels. On average a 10% or more reduction with little loss of quality is possible. There’re two PRs to merge these changes back into the core project but until then, the modified version will be available on GitHub https://github.com/EAddario/llama.cpp/tree/quantize Would love to hear if you can achieve smaller sizes at higher quality!
View all activity

Organizations

None yet

Collections 1