Meta Llama 3
Collection
This collection hosts the transformers and original repos of the Meta Llama 3 and Llama Guard 2 releases
•
5 items
•
Updated
•
739
Sarath Shekkizhar
sarath-shekkizhar
AI & ML interests
None yet
Recent Activity
posted
an
update
about 1 hour ago
Some interesting architectural choices made in Llama 4 models -- were these key to the 10M context? Possibly 🤔
🔍 Takeaways:
🧩 Interleaved Attention without position encoding
- LLaMA 4 removes explicit positional encoding in some attention layers to boost performance on longer contexts.
- The principles here could be similar to the residual connections to facilitate attention to early tokens without positional decay.
⚖️ Scaled Softmax to increase attention at inference time
- The max attention value (output of softmax) decreases as context size increases.
- Llama 4 incorporates a context-size dependent temperature in the softmax function to modify the slope of softmax, allowing the model to focus better on relevant tokens.
- Done only at inference time -- guessing it was more a choice after some observation on eval datasets.
What did you think of these choices?
updated
a Space
9 months ago
tenyx/TenyxChat-7B-v1
updated
a Space
9 months ago
tenyx/TenyxChat-8x7B-v1
Organizations
sarath-shekkizhar's activity
upvoted
a
collection
12 months ago
upvoted
3
collections
about 1 year ago