Article
Welcoming Llama Guard 4 on Hugging Face Hub
ā¢
30
Jared Sulzdorf PRO
jsulz
AI & ML interests
Infrastructure, law, policy
Recent Activity
reacted
to
BramVanroy's
post
with š
about 5 hours ago
š¢š¾ Introducing the Common Crawl Creative Commons Corpus (C5)!
C5 is a large-scale effort to heavily filter web-crawled data, as collected by the non-profit Common Crawl, to only documents that are Creative Commons-licensed such as cc-by-4.0 or public domain cc0. At this stage 150 billion tokens have been collected.
---
š data: https://huggingface.co/datasets/BramVanroy/CommonCrawl-CreativeCommons
š§° software: https://github.com/BramVanroy/CommonCrawl-CreativeCommons
---
</> To build C5, HTML pages are scrutinized and all links (if any) to CC licenses are collected, both in regular hyperlinks as well as in metadata. Additional data fields are included such as "was the license found in the `head`?" or "if multiple licenses were found, do they contradict each other?", which makes further filtering a breeze.
š In this first version of C5, 8 languages are included (Afrikaans, German, English, French, Frysian, Italian, Dutch and Spanish). The language set was limited for two reasons: computational and storage limitations, and a collaboration with GPT-NL, which requested CC data for these languages to train a Dutch-focused, copyright-conscious LLM. In total, this V1 release contains almost 150 thousand documents and 150 billion tokens. This data was not filtered on quality nor deduplicated so that you can decide for yourself how much data to keep. To give some quality indication, a dataset field is present to describe whether a document is included in the FineWeb(-2) datasets, which are of high quality.
š More work needs to be done! Only 7 out of 100+ Common Crawl crawls have been processed so far. That's encouraging because it means there is a lot more Creative Commons data to be collected! But to get there I need help in terms of compute. The current processing was already heavily sponsored by the Flemish Supercomputer but more is needed. If you have the compute available and which to collaborate in an open and transparent manner, please get in touch!
reacted
to
BramVanroy's
post
with ā¤ļø
about 5 hours ago
š¢š¾ Introducing the Common Crawl Creative Commons Corpus (C5)!
C5 is a large-scale effort to heavily filter web-crawled data, as collected by the non-profit Common Crawl, to only documents that are Creative Commons-licensed such as cc-by-4.0 or public domain cc0. At this stage 150 billion tokens have been collected.
---
š data: https://huggingface.co/datasets/BramVanroy/CommonCrawl-CreativeCommons
š§° software: https://github.com/BramVanroy/CommonCrawl-CreativeCommons
---
</> To build C5, HTML pages are scrutinized and all links (if any) to CC licenses are collected, both in regular hyperlinks as well as in metadata. Additional data fields are included such as "was the license found in the `head`?" or "if multiple licenses were found, do they contradict each other?", which makes further filtering a breeze.
š In this first version of C5, 8 languages are included (Afrikaans, German, English, French, Frysian, Italian, Dutch and Spanish). The language set was limited for two reasons: computational and storage limitations, and a collaboration with GPT-NL, which requested CC data for these languages to train a Dutch-focused, copyright-conscious LLM. In total, this V1 release contains almost 150 thousand documents and 150 billion tokens. This data was not filtered on quality nor deduplicated so that you can decide for yourself how much data to keep. To give some quality indication, a dataset field is present to describe whether a document is included in the FineWeb(-2) datasets, which are of high quality.
š More work needs to be done! Only 7 out of 100+ Common Crawl crawls have been processed so far. That's encouraging because it means there is a lot more Creative Commons data to be collected! But to get there I need help in terms of compute. The current processing was already heavily sponsored by the Flemish Supercomputer but more is needed. If you have the compute available and which to collaborate in an open and transparent manner, please get in touch!
Organizations
jsulz's activity
upvoted
an
article
4 days ago
upvoted
a
collection
7 days ago
upvoted
an
article
10 days ago
Article
17 Reasons Why Gradio Isn't Just Another UI Library
ā¢
28
upvoted
a
collection
12 days ago
upvoted
a
collection
13 days ago
upvoted
an
article
16 days ago
Article
Introduction to ggml
ā¢
190
upvoted
a
paper
19 days ago
upvoted
an
article
19 days ago
Article
Cohere on Hugging Face Inference Providers š„
ā¢
124
upvoted
a
collection
29 days ago
upvoted
a
paper
29 days ago
upvoted
2
articles
about 1 month ago
Article
Welcome Llama 4 Maverick & Scout on Hugging Face!
ā¢
142
Article
Training and Finetuning Reranker Models with Sentence Transformers v4
ā¢
125
upvoted
a
paper
about 1 month ago
upvoted
a
collection
about 1 month ago
upvoted
an
article
about 2 months ago
Article
The New and Fresh analytics in Inference Endpoints
ā¢
19
upvoted
2
collections
about 2 months ago