Improve model card: Add description, links, usage, and update metadata (#1)

- Improve model card: Add description, links, usage, and update metadata (7b2d417542461faf6561e4765ae8ba945cd76d18)


Co-authored-by: Niels Rogge <[email protected]>

README.md

@@ -1,11 +1,62 @@
 ---
+datasets:
+- cerebras/SlimPajama-627B
 language:
 - en
+library_name: transformers
+license: mit
+pipeline_tag: text-generation
 tags:
 - text-generation
 - gla
-license: mit
-datasets:
-- cerebras/SlimPajama-627B
-library_name: fla
 ---
+
+# GLA 1.3B-100B: A Hybrid Linear Attention Model
+
+This repository contains the `gla-1.3B-100B` model, a 1.3B parameter variant trained on 100B tokens, which was presented in the paper [A Systematic Analysis of Hybrid Linear Attention](https://huggingface.co/papers/2507.06457).
+
+Transformers face quadratic complexity and memory issues with long sequences, prompting the adoption of linear attention mechanisms. However, linear models often suffer from limited recall performance, leading to hybrid architectures that combine linear and full attention layers. This paper systematically evaluates various linear attention models across generations—vector recurrences to advanced gating mechanisms—both standalone and hybridized. The `gla-1.3B-100B` model is one of 72 models trained and open-sourced to enable this comprehensive analysis. The research highlights that superior standalone linear models do not necessarily excel in hybrids, and emphasizes selective gating, hierarchical recurrence, and controlled forgetting as critical for effective hybrid models. Architectures such as HGRN-2 or GatedDeltaNet with a linear-to-full ratio between 3:1 and 6:1 are recommended for achieving Transformer-level recall efficiently.
+
+## Usage
+
+This model can be easily loaded and used for text generation tasks with the Hugging Face `transformers` library:
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+# Load the tokenizer and model
+model_id = "fla-hub/gla-1.3B-100B"
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
+
+# Example for text generation
+prompt = "Hello, my name is"
+inputs = tokenizer(prompt, return_tensors="pt")
+
+# Generate text
+outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, top_k=50, top_p=0.95, temperature=0.7)
+generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+
+print(generated_text)
+```
+
+## Paper and Citation
+
+If you find this work useful, please consider citing the original paper:
+
+[A Systematic Analysis of Hybrid Linear Attention](https://huggingface.co/papers/2507.06457)
+
+```bibtex
+@article{li2025systematic,
+  title={A Systematic Analysis of Hybrid Linear Attention},
+  author={Li, Tianhong and Deng, Mingyang and He, Kaiming},
+  journal={arXiv preprint arXiv:2507.06457},
+  year={2025},
+}
+```
+
+## Code
+
+The official codebase for the models and research, including training scripts and other checkpoints, can be found on GitHub:
+
+[https://github.com/FLAG-CMU/fla](https://github.com/FLAG-CMU/fla)