winglian commited on
Commit
529c1c5
·
verified ·
1 Parent(s): eb77729

Link to sample YAML

Browse files
Files changed (1) hide show
  1. README.md +5 -2
README.md CHANGED
@@ -96,8 +96,7 @@ license: other
96
  license_name: llama4
97
  ---
98
 
99
-
100
- ## Model Information
101
 
102
  This is a 4-bit Quantized version of this model with the experts broken up and linearized so they play nicely with PEFT/LoRA. To use this with [Axolotl](https://github.com/axolotl-ai-cloud/axolotl), simply include this in your YAML:
103
 
@@ -105,6 +104,10 @@ This is a 4-bit Quantized version of this model with the experts broken up and l
105
  llama4_linearized_experts: true
106
  ```
107
 
 
 
 
 
108
  The Llama 4 collection of models are natively multimodal AI models that enable text and multimodal experiences. These models leverage a mixture-of-experts architecture to offer industry-leading performance in text and image understanding.
109
 
110
  These Llama 4 models mark the beginning of a new era for the Llama ecosystem. We are launching two efficient models in the Llama 4 series, Llama 4 Scout, a 17 billion parameter model with 16 experts, and Llama 4 Maverick, a 17 billion parameter model with 128 experts.
 
96
  license_name: llama4
97
  ---
98
 
99
+ ## Linearized Experts
 
100
 
101
  This is a 4-bit Quantized version of this model with the experts broken up and linearized so they play nicely with PEFT/LoRA. To use this with [Axolotl](https://github.com/axolotl-ai-cloud/axolotl), simply include this in your YAML:
102
 
 
104
  llama4_linearized_experts: true
105
  ```
106
 
107
+ [Sample Axolotl YAML](https://github.com/axolotl-ai-cloud/axolotl/blob/main/examples/llama-4/scout-qlora-fsdp1.yaml)
108
+
109
+ ## Model Information
110
+
111
  The Llama 4 collection of models are natively multimodal AI models that enable text and multimodal experiences. These models leverage a mixture-of-experts architecture to offer industry-leading performance in text and image understanding.
112
 
113
  These Llama 4 models mark the beginning of a new era for the Llama ecosystem. We are launching two efficient models in the Llama 4 series, Llama 4 Scout, a 17 billion parameter model with 16 experts, and Llama 4 Maverick, a 17 billion parameter model with 128 experts.