any hope for running on 256gb ram and 12gb vram ?
hi guys do you think i can run anythink with 256gb ram and 12gb vram ?
100% .theoretically even possible with gpu only and ssd's direct connect
100% .theoretically even possible with gpu only and ssd's direct connect
is it ? can you help me ?
ktransformers creators to answer
recommendation is to try and test yourself
recommendation is to try and test yourself
got it prev i did with small model. very good with ik_llama where i am getting 7.5t/sec with deep seek q2
What exactly do you mean by “direct connect” with the ssd?
pcie
pcie
Any tutorial? I have rn PCI adapter card for NVMe drives. There's nothing on their github about this specifically and in total very few information there, not novice or consumer-friendly. (they should use Ai to write tutorial, merged Qwen with Marco model written great tutorials for merging models process)
Upd: THE ONLY case in all industry (don't know why others hiding so much) of using SSDs for this is Gigabyte with their Ai TOP Utility(very extensive documention), but there SSDs used only for model training not inference. I haven't heard of any other projects last 2 years. That really works? Where tutorials? Where anything?