TinySwallow
Collection
Compact Japanese models trained with "TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models"
ā¢
5 items
ā¢
Updated
ā¢
4
Kaito Sugimoto
kaisugi
AI & ML interests
Japanese LLMs
Recent Activity
upvoted
a
collection
about 8 hours ago
TinySwallow
liked
a model
3 days ago
cyberagent/DeepSeek-R1-Distill-Qwen-32B-Japanese
liked
a model
6 days ago
nu-dialogue/j-moshi-ext
Organizations
kaisugi's activity
upvoted
a
collection
about 8 hours ago
liked
a
model
3 days ago
liked
2
models
6 days ago
liked
a
model
9 days ago
liked
a
Space
10 days ago
reacted to
lianghsun's
post with š
14 days ago
Post
1710
š Let me introduce the work I've done over the past three months: šš¹š®šŗš®-šÆ.š®-š§š®š¶šš®š»-šÆš and šš¹š®šŗš®-šÆ.š®-š§š®š¶šš®š»-šÆš-šš»šššæšš°š, now open-sourced on š¤ Hugging Face.
š¹š¶š®š»š“šµššš»/šš¹š®šŗš®-šÆ.š®-š§š®š¶šš®š»-šÆš: This model is built on top of šŗš²šš®-š¹š¹š®šŗš®/šš¹š®šŗš®-šÆ.š®-šÆš with continual pretraining. The training dataset consists of a mixture of Traditional Chinese and multilingual texts in specific proportions, including 20B tokens of Traditional Chinese text.
š¹š¶š®š»š“šµššš»/šš¹š®šŗš®-šÆ.š®-š§š®š¶šš®š»-šÆš-šš»šššæšš°š: This is a fine-tuned conversational model based on the foundation model.
This Llama-3.2-Taiwan open-source project is currently a one-person effort (yes, I did everything from text preparation ā so exhausting!). If you're interested, feel free to join the Discord server for discussions.
š ±š “š ½š ²š ·š ¼š °šš ŗš øš ½š ¶
The evaluation was conducted using ikala/tmmluplus, though the README page does not yet reflect the latest results. The performance is close to the previous versions, indicating that further improvements might require adding more specialized knowledge in the datasets.
š ° š ²š °š »š » š µš ¾š ššš æš æš ¾šš
If anyone is willing to provide compute resources, it would be greatly appreciated to help this project continue and grow. šŖ
---
šļø Foundation model: lianghsun/Llama-3.2-Taiwan-3B
š¤ Instruction model: lianghsun/Llama-3.2-Taiwan-3B-Instruct
ā” GGUF: lianghsun/Llama-3.2-Taiwan-3B-Instruct-GGUF
š¹š¶š®š»š“šµššš»/šš¹š®šŗš®-šÆ.š®-š§š®š¶šš®š»-šÆš: This model is built on top of šŗš²šš®-š¹š¹š®šŗš®/šš¹š®šŗš®-šÆ.š®-šÆš with continual pretraining. The training dataset consists of a mixture of Traditional Chinese and multilingual texts in specific proportions, including 20B tokens of Traditional Chinese text.
š¹š¶š®š»š“šµššš»/šš¹š®šŗš®-šÆ.š®-š§š®š¶šš®š»-šÆš-šš»šššæšš°š: This is a fine-tuned conversational model based on the foundation model.
This Llama-3.2-Taiwan open-source project is currently a one-person effort (yes, I did everything from text preparation ā so exhausting!). If you're interested, feel free to join the Discord server for discussions.
š ±š “š ½š ²š ·š ¼š °šš ŗš øš ½š ¶
The evaluation was conducted using ikala/tmmluplus, though the README page does not yet reflect the latest results. The performance is close to the previous versions, indicating that further improvements might require adding more specialized knowledge in the datasets.
š ° š ²š °š »š » š µš ¾š ššš æš æš ¾šš
If anyone is willing to provide compute resources, it would be greatly appreciated to help this project continue and grow. šŖ
---
šļø Foundation model: lianghsun/Llama-3.2-Taiwan-3B
š¤ Instruction model: lianghsun/Llama-3.2-Taiwan-3B-Instruct
ā” GGUF: lianghsun/Llama-3.2-Taiwan-3B-Instruct-GGUF
- 4 replies
liked
2
models
16 days ago
liked
a
model
18 days ago
liked
a
model
about 1 month ago
upvoted
2
articles
about 1 month ago
Article
Navigating Korean LLM Research #2: Evaluation Tools
By
ā¢
ā¢
7Article
Navigating Korean LLM Research #1: Models
By
ā¢
ā¢
23
liked
2
models
about 1 month ago
liked
2
models
about 2 months ago
liked
a
Space
about 2 months ago
liked
2
datasets
about 2 months ago