Setup guidance.
I apologize if this is a silly question but I don't have too much experience setting up LLMs locally.
I downloaded this model in LM Studio. And, I guess I am missing some configuration, because it just spits back garbage. If I chat with it, this happens:
You: Are you ready?
Assistant (llama-3.1-8b-claude-3.7-sonnet-reasoning-distilled): onse Rica470470 Rica.gdxižutzeroulousefurt Shore_<?<?姫 �emouthughthton�utzer �vak470ecessoulouse�oulouse\controllers Destruction<? Ricaabinetutzer Rica Ricaonse Shore труemouthonse_<? Rica Shore_<? сбор gloss �<? Sole<? Shore_<?<?furt470priv Chandler<? Superv Rica469_<? fø芳-redux_<?470 SatoshiculateMOOTHecessemouthught Destruction】ughtvak � Rica_<?que673onseربهonse Rica fø Dwight Shore�姫 �agne姫ught<?utzer cubhtonutzer Ricaakest
-- Stop reason: User Stopped --
Similarly when using it with VS Code (CLINE):
-- Task --
What is this std::future post_task_for_gui_thread(Func f)
JMORx
Yikes! You're right. Not sure how that happened!
I'll have it re-run the fine-tuning and upload the model again. Sorry about that! I'll reply here once it's done.
Also, please note: This was an experimental test on my part with the Llama 8B model. I've had much better success with the Gemma 3 models imitating Claude 3.7 Sonnet Reasoning, so if you can use those, I'd recommend them instead:
Gemma 3 12B: https://huggingface.co/reedmayhew/claude-3.7-sonnet-reasoning-gemma3-12B
Working on fine-tuning the Gemma 3 4B soon!
-Reed
Alright, you should be all set! Just delete your local copy of the model and re-download it.
Thanks - Reed
Hi Reed,
Just to confirm, the Q4_0 version is now working properly. Thank you.
The Q8_0 is not, just in case you want to know.
Take care.
Thanks for the update. Not sure why that one is giving me such a hard time!
So frustrating, they were exported at the exact same time with the same llama.cpp instance.
I'll check the file again. - Reed