MythoMax-L2-13B with a support for 8k or 4k context.
Hello, I would like to ask if it posssible to make a MythoMax model with a support for 8k or 4k for context..because I love this model but I think with more context this model gets even great than ever, so I would like to ask if you can make one having this type of stuff, thank you! (and if possible, having support for GGML model too, not only for the normal one)
MythoMax is a LLama 2-based model and has a context of 4k by default. Depending on what you're using to run the model, you can already get 8k context with some settings tweaking as well.
MythoMax is a LLama 2-based model and has a context of 4k by default. Depending on what you're using to run the model, you can already get 8k context with some settings tweaking as well.
all right...in this case, how I can make the model to run using 8k context on Koboldcpp for example? I'm still new at these AI stuff, I saw a 8k model of the MythoMax at the Horde on Agnaistic and I get impressed how much good it is.
in koboldcpp, if using the excutable, there is under tokens a custom ropeconfig , you can either use the linear scaler or ntk aware scaling as follows:
-linear Scaling,for 2x linear scale, set rope scale:0.5 and rope base to :10000, for 4x, use 0.25 10000.
-NTK-Aware Scaling, set rope scale:1.0 32000 for approx 2x scale, or 1.0 82000 for approx 4x .
exemple: i have 4k context limit i (here im using linear scaling) set rope scale:0.5 and rope base to :10000 and now the context limit is 8k
you can find more info here:https://github.com/LostRuins/koboldcpp/wiki