Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindi
Paper
•
2504.06011
•
Published
•
1
ml-fw-prerelease
Enterprise
community
AI & ML interests
None defined yet.
Recent Activity
View all activity
ml-fw-prerelease's activity
alielfilali01
authored
a
paper
9 days ago
Zaid
authored
5
papers
11 days ago
Masader: Metadata Sourcing for Arabic Text and Speech Data Resources
Paper
•
2110.06744
•
Published
Arabic Stable LM: Adapting Stable LM 2 1.6B to Arabic
Paper
•
2412.04277
•
Published
Rephrasing natural text data with different languages and quality levels for Large Language Model pre-training
Paper
•
2410.20796
•
Published
Ashaar: Automatic Analysis and Generation of Arabic Poetry Using Deep Learning Approaches
Paper
•
2307.06218
•
Published
MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs
Paper
•
2505.19800
•
Published
•
1
SivilTaram
authored
a
paper
17 days ago
BramVanroy
posted
an
update
about 1 month ago
Post
3171
📢💾 Introducing the Common Crawl Creative Commons Corpus (C5)!
C5 is a large-scale effort to heavily filter web-crawled data, as collected by the non-profit Common Crawl, to only documents that are Creative Commons-licensed such as cc-by-4.0 or public domain cc0. At this stage 150 billion tokens have been collected.
---
📄 data: BramVanroy/CommonCrawl-CreativeCommons
🧰 software: https://github.com/BramVanroy/CommonCrawl-CreativeCommons
---
</> To build C5, HTML pages are scrutinized and all links (if any) to CC licenses are collected, both in regular hyperlinks as well as in metadata. Additional data fields are included such as "was the license found in the
🌐 In this first version of C5, 8 languages are included (Afrikaans, German, English, French, Frysian, Italian, Dutch and Spanish). The language set was limited for two reasons: computational and storage limitations, and a collaboration with GPT-NL, which requested CC data for these languages to train a Dutch-focused, copyright-conscious LLM. In total, this V1 release contains almost 150 thousand documents and 150 billion tokens. This data was not filtered on quality nor deduplicated so that you can decide for yourself how much data to keep. To give some quality indication, a dataset field is present to describe whether a document is included in the FineWeb(-2) datasets, which are of high quality.
🔍 More work needs to be done! Only 7 out of 100+ Common Crawl crawls have been processed so far. That's encouraging because it means there is a lot more Creative Commons data to be collected! But to get there I need help in terms of compute. The current processing was already heavily sponsored by the Flemish Supercomputer but more is needed. If you have the compute available and which to collaborate in an open and transparent manner, please get in touch!
C5 is a large-scale effort to heavily filter web-crawled data, as collected by the non-profit Common Crawl, to only documents that are Creative Commons-licensed such as cc-by-4.0 or public domain cc0. At this stage 150 billion tokens have been collected.
---
📄 data: BramVanroy/CommonCrawl-CreativeCommons
🧰 software: https://github.com/BramVanroy/CommonCrawl-CreativeCommons
---
</> To build C5, HTML pages are scrutinized and all links (if any) to CC licenses are collected, both in regular hyperlinks as well as in metadata. Additional data fields are included such as "was the license found in the
head?" or "if multiple licenses were found, do they contradict each other?", which makes further filtering a breeze. 🌐 In this first version of C5, 8 languages are included (Afrikaans, German, English, French, Frysian, Italian, Dutch and Spanish). The language set was limited for two reasons: computational and storage limitations, and a collaboration with GPT-NL, which requested CC data for these languages to train a Dutch-focused, copyright-conscious LLM. In total, this V1 release contains almost 150 thousand documents and 150 billion tokens. This data was not filtered on quality nor deduplicated so that you can decide for yourself how much data to keep. To give some quality indication, a dataset field is present to describe whether a document is included in the FineWeb(-2) datasets, which are of high quality.
🔍 More work needs to be done! Only 7 out of 100+ Common Crawl crawls have been processed so far. That's encouraging because it means there is a lot more Creative Commons data to be collected! But to get there I need help in terms of compute. The current processing was already heavily sponsored by the Flemish Supercomputer but more is needed. If you have the compute available and which to collaborate in an open and transparent manner, please get in touch!
-
- 1 reply
alielfilali01
posted
an
update
about 1 month ago
Post
614
Great efforts from
@AtlasIA
folks to adapt text2image models (ghibli style) for Moroccan Context
Read the blog is here : https://huggingface.co/blog/atlasia/creating-your-custom-ghibli-text-to-image-model
Read the blog is here : https://huggingface.co/blog/atlasia/creating-your-custom-ghibli-text-to-image-model
jsaizant
authored
a
paper
about 1 month ago
nouamanetazi
authored
a
paper
about 2 months ago
SivilTaram
authored
6
papers
2 months ago
Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows
Paper
•
2411.07763
•
Published
•
2
When Attention Sink Emerges in Language Models: An Empirical View
Paper
•
2410.10781
•
Published
Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs
Paper
•
2502.12982
•
Published
•
17
Predictive Data Selection: The Data That Predicts Is the Data That Teaches
Paper
•
2503.00808
•
Published
•
57
Scaling up Masked Diffusion Models on Text
Paper
•
2410.18514
•
Published
SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
Paper
•
2503.18892
•
Published
•
30
SivilTaram
authored
a
paper
3 months ago
Post
2831
We introduce FAT5 (Flash Attention T5) ⚡
An implementation of T5 in PyTorch with UL2 objective optimized for GPGPU for both training and inference thanks to 13 different optimizations.
The main one is that we have designed a CUDA kernel to expand the Flash Attention by @tridao with RPE biases and supports other PE such as RoPE, ALiBi or FIRE.
The result kernel is 2 times faster than a SPDA implementation.
We also use Triton kernels to optimize certain parts of the architecture, such as the cross-entropy and RMSNorm layer.
The various kernels have been carefully built to be compatible with BF16 and torch.compile to go even faster and achieve efficient pretraining.
All other optimizations are described in a 📝 subsequent blog post available on @huggingface 🤗: CATIE-AQ/FAT5-report.
This methodology enabled us to efficiently pretrain as a proof of concept a FAT5 with 147M parameters in French in a reasonable time (1,461H for 419B tokens), with limited resources (1 A100 i.e. a computational budget of ~ €1,900) and a low carbon footprint (13.5kg eq CO2).
The model's weights are also available on Hugging Face: CATIE-AQ/FAT5-small.
Not very useful in practice, it's a PoC and not an instructed model (it's planned for later).
All the code is available on GitHub if you want to pretrain your own model in your own language or for a specific domain: https://github.com/catie-aq/flashT5 ⭐
Ending by indicating that was a joint project with @BorisAlbar at hf.co/CATIE-AQ.
An implementation of T5 in PyTorch with UL2 objective optimized for GPGPU for both training and inference thanks to 13 different optimizations.
The main one is that we have designed a CUDA kernel to expand the Flash Attention by @tridao with RPE biases and supports other PE such as RoPE, ALiBi or FIRE.
The result kernel is 2 times faster than a SPDA implementation.
We also use Triton kernels to optimize certain parts of the architecture, such as the cross-entropy and RMSNorm layer.
The various kernels have been carefully built to be compatible with BF16 and torch.compile to go even faster and achieve efficient pretraining.
All other optimizations are described in a 📝 subsequent blog post available on @huggingface 🤗: CATIE-AQ/FAT5-report.
This methodology enabled us to efficiently pretrain as a proof of concept a FAT5 with 147M parameters in French in a reasonable time (1,461H for 419B tokens), with limited resources (1 A100 i.e. a computational budget of ~ €1,900) and a low carbon footprint (13.5kg eq CO2).
The model's weights are also available on Hugging Face: CATIE-AQ/FAT5-small.
Not very useful in practice, it's a PoC and not an instructed model (it's planned for later).
All the code is available on GitHub if you want to pretrain your own model in your own language or for a specific domain: https://github.com/catie-aq/flashT5 ⭐
Ending by indicating that was a joint project with @BorisAlbar at hf.co/CATIE-AQ.
SivilTaram
authored
a
paper
4 months ago