UPDATE, PATRIOTS! 🇺🇸🦅
Well, I spent some time on that "desuarchive /g/" thing Timmy mentioned (still not sure what all these slashes and letters mean! 😂), and guess what?! It seems "/lmg/" is NOT gone for good! Hallelujah! 🙏
Apparently, according to what I read there, it kinda split up into THREE different places! One of them is like "4chan, but GAY" (kids these days, right?! 😂 or was it DOT gay? dunno remember). Another one they mentioned is called "8chan moe" - now, I gotta say, that doesn't sound like the good ol' 8chan from back in my day on the NET, but who knows! And the last one they talked about was "erischan org". Apparently, it's run by folks who are into this "Discordianism" thing, which sounds like some kinda… well, a bit like a religious group, but from what I gather, they are all good folks and know how to not take everything SO seriously! Seems like a friendly atmosphere there, which is always a good thing! 👍
Thanks again for all your help, you young whippersnappers are alright! 😉
God bless and keep fighting the good fight! 🇺🇸
God bless America 🇺🇸
#WWG1WGA
John, thanks for the info, appreciate it! "Reddit," huh? Timmy mentioned something about that place, but said it was full of... well, let's just say folks who aren't always on the same page as yours truly.
This "gone forever" business is a real shame. You wouldn't happen to know if those "/lmg/" folks maybe went somewhere else, would ya? Timmy mentioned something about "alt chans" and "disc cords"... are those like some kind of secret handshake clubs on the internet?
Anyways, thanks again for the help, John. Appreciate a fellow patriot helping an old-timer out.
God bless America 🇺🇸
#WWG1WGA
3% of all models or only LLMs?
Got it, so you try all major prompt formats (Alpaca, Vicuna, ChatML, Mistral, etc)_and get the best result?
Usually 2 is enough, official and alpaca. In some cases I go further and try chatting to it in vicuna-style format or a modification of official format.
Full deterministic? Disable all samplers and temp 0?
Yep. No samplers. Temp 0. This caused some models to fail in the past due to repetition issues, but modern models are okay with no rep penalty.
When you are evaluating, are you using vanilla prompt and temp 1?
I use whichever prompt format works best, usually the official, but some models(like Mistral family) perform better with other prompt formats. For evaluation of Largestral I used Alpaca, since it worked better than the official format. I always use deterministic settings in my evaluations.
It performed quite well on my bench. Doesn't mean much since we are reaching the post-benchmark era.
@jukofyork Quite a difference indeed. What was Zuck thinking when they filtered base model data? Was he thinking? (I doubt it.) Anyway, it's nice to have competition.
Remain calm.
Local models endure.
Cohere lives.
The finetuners shall endure.
There is much to be done.
This seems to agree it's a non-starter for creative-writing purposes:
I know that this benchmark is not very reliable(CR+ below 14b qwen), but how did Meta manage to make L3.1 70b perform worse than L3 70b? How? How much data did they filter?
https://ai.meta.com/research/publications/the-llama-3-herd-of-models/
We create our dataset for language model pre-training from a variety of data sources containing knowledge
until the end of 2023. We apply several de-duplication methods and data cleaning mechanisms on each data
source to obtain high-quality tokens. We remove domains that contain large amounts of personally identifiable
information (PII), and domains with known adult content
We use “dirty word” counting (Raffel et al., 2020) to filter out adult websites that are not covered by
domain block lists.
Did I just download 760GB for nothing?
