Hugging Face's logo
Sarath Shekkizhar's picture
7 4 22

Sarath Shekkizhar

sarath-shekkizhar
Bossmarc747's profile picture tasmani's profile picture pkcons9's profile picture
Ā·
https://shekkizh.github.io/

AI & ML interests

None yet

Recent Activity

posted an update about 1 hour ago
Some interesting architectural choices made in Llama 4 models -- were these key to the 10M context? Possibly šŸ¤” šŸ” Takeaways: šŸ§© Interleaved Attention without position encoding - LLaMA 4 removes explicit positional encoding in some attention layers to boost performance on longer contexts. - The principles here could be similar to the residual connections to facilitate attention to early tokens without positional decay. āš–ļø Scaled Softmax to increase attention at inference time - The max attention value (output of softmax) decreases as context size increases. - Llama 4 incorporates a context-size dependent temperature in the softmax function to modify the slope of softmax, allowing the model to focus better on relevant tokens. - Done only at inference time -- guessing it was more a choice after some observation on eval datasets. What did you think of these choices?
updated a Space 9 months ago
tenyx/TenyxChat-7B-v1
updated a Space 9 months ago
tenyx/TenyxChat-8x7B-v1
View all activity

Organizations

Salesforce's profile picture Tenyx's profile picture Blog-explorers's profile picture ZeroGPU Explorers's profile picture Social Post Explorers's profile picture Hugging Face Discord Community's profile picture

sarath-shekkizhar's activity

liked 4 datasets 10 months ago
liked a dataset 11 months ago
liked 3 Spaces 11 months ago
liked a model 12 months ago
liked a Space 12 months ago
liked a model 12 months ago
liked 2 Spaces about 1 year ago
liked 2 models about 1 year ago
liked a dataset about 1 year ago
liked 2 Spaces about 1 year ago
liked 2 datasets about 1 year ago