---
title: README
emoji: 👀
colorFrom: purple
colorTo: red
sdk: static
pinned: false
---

## What is this?

A dataprocessing pipeline that uses huggingface datsets as intermediate data store.

Metadata are designed to be updated like a DAG, where some depends on others. 

Workflows are gradually being built over time and maybe we'll see hundreds of data repos one day.

## How do I use it?

To load files in local, Huggingface as well as S3 a tool is being developed in progress.

![image/png](https://cdn-uploads.huggingface.co/production/uploads/636982a164aad59d4d42714b/hF8K4nVQUmJXNHs6NN8_8.png)