Spaces:
Running
on
CPU Upgrade
Benchmarking Stability Japanese Stable LM variants?
I think it would be valuable to add Stability AI's suite of Japanese fine-tuned models (https://huggingface.co/collections/stabilityai/japanese-stable-lm-654063a381a8731a1c0f13cc ) to the leaderboard (or at least some of them). They are Japanese specific models released by a major reputable AI lab, I think that makes them highly relevant. However, these models are either gated, or require remote code execution, so they can not be submitted through the leaderboard submission interface. Would anyone who runs llm-jp be willing to add submissions for these models manually?
Thanks
Hello Eric !
Thank you for your interest in the Open Japanese LLM Leaderboard by LLM-jp. We are currently working on the version 2 of the leaderboard. The backend is based on vLLM, so we were bound to the supported models of vLLM. As you said, many LLMs are gated, modify architecture, or required code execution, etc. Hopefully, the recent collaboration between Transformers and vLLM will help us to enlarge our evaluation. Thank you for your patience ! : )
Best,
Akim