4.55BPW 24k context on 48GB of VRAM
#1
by
Adzeiros
- opened
So I downloaded the 4.55BPW version, but when loading it in TabbyAPI, using 24576 as the context length, and 512 chunk size it still goes OOM... Or did you mean flat 24000?
Are your GPU’s dedicated?
If you are sharing VRAM with your system or another program it will not fit 24k.
Unfortunately, it appears you are correct in this instance. I used a vram calculator and it looks like it was wrong..
won’t be doing that again.
Thank you for letting me know.
Updated the readme, removed the context from it.
FrenzyBiscuit
changed discussion status to
closed