Duplicate from BlinkDL/rwkv-7-world

Co-authored-by: BlinkDL <[email protected]>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,104 @@
+---
+language:
+- en
+- zh
+- fr
+- es
+- de
+- pt
+- ru
+- it
+- ja
+- ko
+- vi
+- ar
+tags:
+- pytorch
+- text-generation
+- causal-lm
+- rwkv
+license: apache-2.0
+datasets:
+- HuggingFaceFW/fineweb-edu
+- mlfoundations/dclm-baseline-1.0
+- cerebras/SlimPajama-627B
+- EleutherAI/pile
+- bigcode/starcoderdata
+- oscar-corpus/OSCAR-2301
+---
+
+# RWKV-7 World
+
+Use rwkv pip package 0.8.28+ for RWKV-7 inference: https://pypi.org/project/rwkv/
+
+Evals and more information: https://rwkv.com/
+
+For developers: https://github.com/BlinkDL/RWKV-LM
+
+Chat demo: https://github.com/BlinkDL/ChatRWKV/blob/main/API_DEMO_CHAT.py
+
+MMLU eval: https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/rwkv_mmlu_eval.py
+
+rwkv7-v3-2.9b 54.56% (rwkv6-v2.1-3.1b 32.38%)
+
+rwkv7-v3-1.5b 44.84% (rwkv6-v2.1-1.6b 26.34%)
+
+0.1B = L12 D768 // 0.4B = L24 D1024 // 1.5B = L24 D2048 // ~3B = L32 D2560 // ~7B = L32 D4096
+
+## Model Description
+
+RWKV-7 trained on 100+ world languages (80% English, 10% multilang, 10% code).
+
+World-v3 = 3.1T tokens
+
+World-v2.9 = subsampled 2T tokens
+
+World-v2.8 = subsampled 1T tokens
+
+Recommended fine-tuning format (use \n for newlines):
+```
+User: xxxxxxxxxxxxxxx
+
+Assistant: xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+
+User: xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+
+Assistant: xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+xxxxxxxxxxxxxxx
+```
+
+A good chat prompt (better replace \n\n in xxx to \n, such that there will never be extra \n\n in response):
+```
+User: hi
+
+Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
+
+User: xxx
+
+Assistant:
+```
+QA prompt (better replace \n\n in xxx to \n, such that there will never be extra \n\n in response):
+```
+Question: xxx
+
+Answer:
+```
+and
+```
+Instruction: xxx
+
+Input: xxx
+
+Response:
+```
+
+!!! There should not be any space after your final ":" or you will upset the tokenizer and see non-English reponse !!!
+
+!!! There should not be any space after your final ":" or you will upset the tokenizer and see non-English reponse !!!
+
+!!! There should not be any space after your final ":" or you will upset the tokenizer and see non-English reponse !!!

RWKV-x070-World-0.1B-v2.8-20241210-ctx4096.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:60c98129b9529963bff2c164b8ab4bd17c19332ae06dc2dcae32aa3a3739295a
+size 382195690

RWKV-x070-World-0.4B-v2.9-20250107-ctx4096.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c08cf602b33e59a5717b8f0cafaf7f04c50c1c67166e477aa2f47c5ca180da4a
+size 901794466

RWKV-x070-World-1.5B-v3-20250127-ctx4096.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23e13ff62ac6f80b28910ffc230324939e80ddb00b98f892a5951c0da071e700
+size 3055062345

RWKV-x070-World-2.9B-v3-20250211-ctx4096.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d7901b91ef1646115db03d03d171370bca454061ad426a1256fda9e1c428cf8
+size 5895796446