Sam Gillham PRO

I'm attempting to use a 7B LLM, Llama in this case, with an embedding head stuck on the end instead of the lm_head. I used an LLM to rank a ton of randomly selected pairs of papers based on if they have good connections, and trained the embedding head on triplets mined from those ranked pairs.

The idea is for the embedding head to learn to align features from paper abstracts that complement each other.

this is the first version and yeah, I'm not overly impressed. I think I'm seeing results that kinda vibe with the concept sometimes, but I think the ranking criteria for the dataset were a bit loose. I'm going to try making a new dataset with better, more strict, more nuanced criteria and train a second version of the model from that.

Thanks for letting me know. I've fixed the issue. Feel free to try again.

I think we can extract that Harvard knowledge and distribute it in the form of properly open models. Get them chatting with our LLMs and train on the collected knowledge. Mwahaha!

They do it to us, after all.

5090 because I'm not a millionaire.

it's pretty exciting to see the newer, more powerful hardware coming down the pipeline. with better hardware becoming more prolific I hope we can see expansions of concepts like BOINC and Folding@Home and newer ideas as well. It's impossible for you or I to compete with BigAI, but with federated learning techniques and enough people I think it can be done, at least "good enough". It sounds like the kind of thing you might be interested in, yeah?