Update README.md

README.md

@@ -11,11 +11,9 @@ tags:
 - gla
 ---
 
-# GLA 1.3B-100B: A Hybrid Linear Attention Model
+# GLA 1.3B-100B
 
-This repository contains the `gla-1.3B-100B` model, a 1.3B parameter variant trained on 100B tokens, which was presented in the paper [A Systematic Analysis of Hybrid Linear Attention](https://huggingface.co/papers/2507.06457).
-
-Transformers face quadratic complexity and memory issues with long sequences, prompting the adoption of linear attention mechanisms. However, linear models often suffer from limited recall performance, leading to hybrid architectures that combine linear and full attention layers. This paper systematically evaluates various linear attention models across generations—vector recurrences to advanced gating mechanisms—both standalone and hybridized. The `gla-1.3B-100B` model is one of 72 models trained and open-sourced to enable this comprehensive analysis. The research highlights that superior standalone linear models do not necessarily excel in hybrids, and emphasizes selective gating, hierarchical recurrence, and controlled forgetting as critical for effective hybrid models. Architectures such as HGRN-2 or GatedDeltaNet with a linear-to-full ratio between 3:1 and 6:1 are recommended for achieving Transformer-level recall efficiently.
+This repository contains the `gla-1.3B-100B` model, a 1.3B parameter variant trained on 100B tokens, which was presented in the paper [Gated Linear Attention Transformers with Hardware-Efficient Training](https://huggingface.co/papers/2312.06635).
 
 ## Usage
 
@@ -44,14 +42,14 @@ print(generated_text)
 
 If you find this work useful, please consider citing the original paper:
 
-[A Systematic Analysis of Hybrid Linear Attention](https://huggingface.co/papers/2507.06457)
+[Gated Linear Attention Transformers with Hardware-Efficient Training](https://huggingface.co/papers/2507.06457)
 
 ```bibtex
 @article{li2025systematic,
-  title={A Systematic Analysis of Hybrid Linear Attention},
-  author={Li, Tianhong and Deng, Mingyang and He, Kaiming},
-  journal={arXiv preprint arXiv:2507.06457},
-  year={2025},
+  title={Gated Linear Attention Transformers with Hardware-Efficient Training},
+  author={Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, Yoon Kim},
+  journal={arXiv preprint arXiv:2312.06635},
+  year={2023},
 }
 ```
 
@@ -59,4 +57,4 @@ If you find this work useful, please consider citing the original paper:
 
 The official codebase for the models and research, including training scripts and other checkpoints, can be found on GitHub:
 
-[https://github.com/FLAG-CMU/fla](https://github.com/FLAG-CMU/fla)
+[https://github.com/fla-org/flash-linear-attention](https://github.com/fla-org/flash-linear-attention)