Is it possible to fine-tune gemma 3 in a context beyond 131k?
#3
by
dophys
- opened
Hello. Gemma 3 claims to support only 131K long sequences ("max_position_embeddings": 131072,). Can we fine-tune it again to increase the length of contexts it supports? I vaguely remember that there seems to be some technology like rope that can support infinitely long sequences. If so, for unsloth I'm not sure how to go about it? Thank you for your work and it would be great if you could answer this question again.