Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard
•
30
Ali El Filali
alielfilali01
AI & ML interests
AI Psychometrician ? | NLP (mainly for Arabic) | Other interests include Reinforcement Learning and Cognitive sciences among others
Recent Activity
upvoted
an
article
2 days ago
Train 400x faster Static Embedding Models with Sentence Transformers
commented on
an
article
2 days ago
CO₂ Emissions and Models Performance: Insights from the Open LLM Leaderboard
upvoted
an
article
2 days ago
CO₂ Emissions and Models Performance: Insights from the Open LLM Leaderboard
Articles
Organizations
alielfilali01's activity
upvoted
an
article
2 days ago
Article
Train 400x faster Static Embedding Models with Sentence Transformers
•
101
commented on
CO₂ Emissions and Models Performance: Insights from the Open LLM Leaderboard
2 days ago
Inspiring piece
@alozowski
! I want to ask if what you guys report is the KG-CO2 per model or ration of KG-CO2:#params ?
Even if it is easily inferenced info, I believe both infos are equally important to display ... or i might be missing something here ?
upvoted
an
article
2 days ago
Article
CO₂ Emissions and Models Performance: Insights from the Open LLM Leaderboard
•
14
updated
a
Space
2 days ago
updated
a
dataset
3 days ago
liked
2
Spaces
3 days ago
liked
2
models
3 days ago
upvoted
a
paper
3 days ago
liked
a
dataset
8 days ago
updated
a
dataset
8 days ago
liked
a
Space
8 days ago
liked
2
Spaces
9 days ago
upvoted
a
paper
9 days ago
liked
2
Spaces
11 days ago
reacted to
MohamedRashad's
post with ❤️
11 days ago
Post
1825
The winners of Best Paper Award in NeurIPs2024 (FoundationVision)
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale
Prediction (2404.02905) has just released a new paper called infinty:
Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2412.04431)
And i managed to build a space for it so anyone can try it out: MohamedRashad/Infinity
The idea of a text to image model using autoregressive archticture is quite interesting in my opinion.
Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2412.04431)
And i managed to build a space for it so anyone can try it out: MohamedRashad/Infinity
The idea of a text to image model using autoregressive archticture is quite interesting in my opinion.
posted
an
update
11 days ago
Post
1791
3C3H AraGen Leaderboard welcomes today
deepseek-ai/DeepSeek-V3 and 12 other models (including the late gpt-3.5 💀) to the ranking of best LLMs in Arabic !
Observations:
- DeepSeek-v3 ranked 3rd and only Open model among the top 5 !
- A 14B open model ( Qwen/Qwen2.5-14B-Instruct) outperforms gpt-3.5-turbo-0125 (from last year). This shows how much we came in advancing and supporting Arabic presence within the LLM ecosystem !
- Contrary to what observed in likelihood-acc leaderboards (like OALL/Open-Arabic-LLM-Leaderboard) further finetuned models like maldv/Qwentile2.5-32B-Instruct actually decreased the performance compared to the original model Qwen/Qwen2.5-32B-Instruct.
It's worth to note that the decrease is statiscally insignificant which imply that at best, the out-domain finetuning do not really hurts the model original capabilities acquired during pretraining.
Previous work addressed this (finetuning VS pretraining) but more investigation in this regard is required (any PhDs here ? This could be your question ...)
Check out the latest rankings: inceptionai/AraGen-Leaderboard
Observations:
- DeepSeek-v3 ranked 3rd and only Open model among the top 5 !
- A 14B open model ( Qwen/Qwen2.5-14B-Instruct) outperforms gpt-3.5-turbo-0125 (from last year). This shows how much we came in advancing and supporting Arabic presence within the LLM ecosystem !
- Contrary to what observed in likelihood-acc leaderboards (like OALL/Open-Arabic-LLM-Leaderboard) further finetuned models like maldv/Qwentile2.5-32B-Instruct actually decreased the performance compared to the original model Qwen/Qwen2.5-32B-Instruct.
It's worth to note that the decrease is statiscally insignificant which imply that at best, the out-domain finetuning do not really hurts the model original capabilities acquired during pretraining.
Previous work addressed this (finetuning VS pretraining) but more investigation in this regard is required (any PhDs here ? This could be your question ...)
Check out the latest rankings: inceptionai/AraGen-Leaderboard