metadata
title: README
emoji: 👀
colorFrom: purple
colorTo: red
sdk: static
pinned: false
What is this?
A dataprocessing pipeline that uses huggingface datsets as intermediate data store.
Metadata are designed to be updated like a DAG, where some depends on others.
Workflows are gradually being built over time and maybe we'll see hundreds of data repos one day.
How do I use it?
To load files in local, Huggingface as well as S3 a tool is being developed in progress.
