ZeusLabs/Chronos-Platinum-72B
Text Generation
•
Updated
•
75
•
11
Zeus Labs
community
AI & ML interests
None defined yet.
ZeusLabs's activity
Special tokens for instruction template
1
#3 opened 7 months ago
by
Weker
elinas
updated
a
model
3 months ago
elinas
updated
a
model
4 months ago
Post
2208
We conducted an experiment in an effort to revive LLaMA 1 33B as it had unique prose and a lack of "GPT-isms" and "slop" in its pretraining data, as well as being one of the favorites at the time. With multiple finetune runs, we were able to extend the model from it's pretrained base of 2048 to ~12,000 tokens adding approx. 500M tokens in the process. The effective length is 16,384 but it's better to keep it on the lower range. It writes well and in multiple formats. In the future, we have some ideas like implementing GQA. Please take a look and we would love to hear your feedback!
ZeusLabs/Chronos-Divergence-33B
ZeusLabs/Chronos-Divergence-33B
elinas
updated
a
model
4 months ago
elinas
updated
a
Space
4 months ago
Steelskull
updated
a
model
6 months ago
Fizzarolli
posted
an
update
8 months ago
Post
2026
hi everyone!
i wanted to share an experiment i did with upcycling phi-3 mini into an moe recently.
while benchmarks are definitely within a margin of error and they performed similarly, i think it's an interesting base to try and see if you can improve phi's performance! (maybe looking into HuggingFaceFW/fineweb-edu could be interesting, i also left some other notes if anyone with more compute access wants to try it themselves)
check it out! Fizzarolli/phi3-4x4b-v1
i wanted to share an experiment i did with upcycling phi-3 mini into an moe recently.
while benchmarks are definitely within a margin of error and they performed similarly, i think it's an interesting base to try and see if you can improve phi's performance! (maybe looking into HuggingFaceFW/fineweb-edu could be interesting, i also left some other notes if anyone with more compute access wants to try it themselves)
check it out! Fizzarolli/phi3-4x4b-v1
Fizzarolli
posted
an
update
8 months ago
Post
2454
Is anyone looking into some sort of decentralized/federated dataset generation or classification by humans instead of synthetically?
From my experience with trying models, a *lot* of modern finetunes are trained on what amounts to, in essence, GPT-4 generated slop that makes everything sound like a rip-off GPT-4 (refer to i.e. the Dolphin finetunes). I have a feeling that this is a lot of the reason people haven't been quite as successful as Meta's instruct tunes of Llama 3.
From my experience with trying models, a *lot* of modern finetunes are trained on what amounts to, in essence, GPT-4 generated slop that makes everything sound like a rip-off GPT-4 (refer to i.e. the Dolphin finetunes). I have a feeling that this is a lot of the reason people haven't been quite as successful as Meta's instruct tunes of Llama 3.