NousResearch/Hermes-2-Theta-Llama-3-8B
Text Generation
•
8B
•
Updated
•
10.8k
•
•
203
kon g
dolemole
·
AI & ML interests
None yet
Recent Activity
new activity
about 1 month ago
calcuis/chatterbox-gguf:Just wanted to say thanks Calcuis
liked
a model
about 1 year ago
NousResearch/Hermes-2-Theta-Llama-3-8B
liked
a model
about 1 year ago
BadToBest/EchoMimic
Organizations
None yet
Just wanted to say thanks Calcuis
👍
1
27
#1 opened about 1 month ago
by
Winnougan
liked
2
models
about 1 year ago
reacted to
akhaliq's
post with ❤️
over 1 year ago
Post
Simple and Scalable Strategies to Continually Pre-train Large Language Models
Simple and Scalable Strategies to Continually Pre-train Large Language Models (2403.08763)
Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes available. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by final loss and language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (EnglishrightarrowEnglish) and a stronger distribution shift (EnglishrightarrowGerman) at the 405M parameter model scale with large dataset sizes (hundreds of billions of tokens). Selecting the weak but realistic shift for larger-scale experiments, we also find that our continual learning strategies match the re-training baseline for a 10B parameter LLM. Our results demonstrate that LLMs can be successfully updated via simple and scalable continual learning strategies, matching the re-training baseline using only a fraction of the compute. Finally, inspired by previous work, we propose alternatives to the cosine learning rate schedule that help circumvent forgetting induced by LR re-warming and that are not bound to a fixed token budget.
Simple and Scalable Strategies to Continually Pre-train Large Language Models (2403.08763)
Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes available. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by final loss and language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (EnglishrightarrowEnglish) and a stronger distribution shift (EnglishrightarrowGerman) at the 405M parameter model scale with large dataset sizes (hundreds of billions of tokens). Selecting the weak but realistic shift for larger-scale experiments, we also find that our continual learning strategies match the re-training baseline for a 10B parameter LLM. Our results demonstrate that LLMs can be successfully updated via simple and scalable continual learning strategies, matching the re-training baseline using only a fraction of the compute. Finally, inspired by previous work, we propose alternatives to the cosine learning rate schedule that help circumvent forgetting induced by LR re-warming and that are not bound to a fixed token budget.
Can I use this in ComfyUI or Auto1111 SD WebUI?
4
#8 opened over 1 year ago
by
RageshAntony
upvoted
2
papers
over 1 year ago
HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesis
Paper
•
2311.12454
•
Published
•
31
CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model
Paper
•
2305.06908
•
Published
•
6
liked
a
model
over 1 year ago
upvoted
2
papers
over 1 year ago
updated
a
collection
over 1 year ago
upvoted
a
collection
over 1 year ago
updated
a
collection
almost 2 years ago
liked
2
models
almost 2 years ago
liked
a
dataset
almost 2 years ago