Add tokenizer.model for compatibility with older Transformers versions
Browse filesSome environments require tokenizer.model (SentencePiece).
Adding it here to improve compatibility while tokenizer.json remains the default.
- .gitattributes +1 -0
- tokenizer.model +3 -0
.gitattributes
CHANGED
|
@@ -2,3 +2,4 @@
|
|
| 2 |
*.png filter=lfs diff=lfs merge=lfs -text
|
| 3 |
*.json filter=lfs diff=lfs merge=lfs -text
|
| 4 |
*.jinja filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 2 |
*.png filter=lfs diff=lfs merge=lfs -text
|
| 3 |
*.json filter=lfs diff=lfs merge=lfs -text
|
| 4 |
*.jinja filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
tokenizer.model filter=lfs diff=lfs merge=lfs -text
|
tokenizer.model
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
|
| 3 |
+
size 4689074
|