Link to sample YAML
Browse files
README.md
CHANGED
@@ -96,8 +96,7 @@ license: other
|
|
96 |
license_name: llama4
|
97 |
---
|
98 |
|
99 |
-
|
100 |
-
## Model Information
|
101 |
|
102 |
This is a 4-bit Quantized version of this model with the experts broken up and linearized so they play nicely with PEFT/LoRA. To use this with [Axolotl](https://github.com/axolotl-ai-cloud/axolotl), simply include this in your YAML:
|
103 |
|
@@ -105,6 +104,10 @@ This is a 4-bit Quantized version of this model with the experts broken up and l
|
|
105 |
llama4_linearized_experts: true
|
106 |
```
|
107 |
|
|
|
|
|
|
|
|
|
108 |
The Llama 4 collection of models are natively multimodal AI models that enable text and multimodal experiences. These models leverage a mixture-of-experts architecture to offer industry-leading performance in text and image understanding.
|
109 |
|
110 |
These Llama 4 models mark the beginning of a new era for the Llama ecosystem. We are launching two efficient models in the Llama 4 series, Llama 4 Scout, a 17 billion parameter model with 16 experts, and Llama 4 Maverick, a 17 billion parameter model with 128 experts.
|
|
|
96 |
license_name: llama4
|
97 |
---
|
98 |
|
99 |
+
## Linearized Experts
|
|
|
100 |
|
101 |
This is a 4-bit Quantized version of this model with the experts broken up and linearized so they play nicely with PEFT/LoRA. To use this with [Axolotl](https://github.com/axolotl-ai-cloud/axolotl), simply include this in your YAML:
|
102 |
|
|
|
104 |
llama4_linearized_experts: true
|
105 |
```
|
106 |
|
107 |
+
[Sample Axolotl YAML](https://github.com/axolotl-ai-cloud/axolotl/blob/main/examples/llama-4/scout-qlora-fsdp1.yaml)
|
108 |
+
|
109 |
+
## Model Information
|
110 |
+
|
111 |
The Llama 4 collection of models are natively multimodal AI models that enable text and multimodal experiences. These models leverage a mixture-of-experts architecture to offer industry-leading performance in text and image understanding.
|
112 |
|
113 |
These Llama 4 models mark the beginning of a new era for the Llama ecosystem. We are launching two efficient models in the Llama 4 series, Llama 4 Scout, a 17 billion parameter model with 16 experts, and Llama 4 Maverick, a 17 billion parameter model with 128 experts.
|