Update README.md
Browse files
README.md
CHANGED
|
@@ -7,4 +7,14 @@ sdk: static
|
|
| 7 |
pinned: false
|
| 8 |
---
|
| 9 |
|
| 10 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
pinned: false
|
| 8 |
---
|
| 9 |
|
| 10 |
+
## What is this?
|
| 11 |
+
|
| 12 |
+
A dataprocessing pipeline that uses huggingface datsets as intermediate data store.
|
| 13 |
+
|
| 14 |
+
Metadata are designed to be updated like a DAG, where some depends on others.
|
| 15 |
+
|
| 16 |
+
Workflows are gradually being built over time and maybe we'll see hundreds of data repos one day.
|
| 17 |
+
|
| 18 |
+
## How do I use it?
|
| 19 |
+
|
| 20 |
+
To load files in local, Huggingface as well as S3 a tool is being developed in progress.
|