Upload Llama4ForCausalLM

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

config.json

@@ -0,0 +1,152 @@
+{
+  "architectures": [
+    "Llama4ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_chunk_size": 8192,
+  "attention_dropout": 0.0,
+  "attn_scale": 0.1,
+  "attn_temperature_tuning": 4,
+  "bos_token_id": 200000,
+  "cache_implementation": "hybrid",
+  "eos_token_id": [
+    200001,
+    200007,
+    200008
+  ],
+  "floor_scale": 8192,
+  "for_llm_compressor": false,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "interleave_moe_layer_step": 1,
+  "intermediate_size": 8192,
+  "intermediate_size_mlp": 16384,
+  "max_position_embeddings": 262144,
+  "model_type": "llama4_text",
+  "moe_layers": [
+    0,
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    10,
+    11,
+    12,
+    13,
+    14,
+    15,
+    16,
+    17,
+    18,
+    19,
+    20,
+    21,
+    22,
+    23,
+    24,
+    25,
+    26,
+    27,
+    28,
+    29,
+    30,
+    31,
+    32,
+    33,
+    34,
+    35,
+    36,
+    37,
+    38,
+    39,
+    40,
+    41,
+    42,
+    43,
+    44,
+    45,
+    46,
+    47
+  ],
+  "no_rope_layers": [
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0
+  ],
+  "num_attention_heads": 40,
+  "num_experts_per_tok": 1,
+  "num_hidden_layers": 48,
+  "num_key_value_heads": 8,
+  "num_local_experts": 16,
+  "output_router_logits": false,
+  "pad_token_id": 200018,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": {
+    "factor": 8.0,
+    "high_freq_factor": 4.0,
+    "low_freq_factor": 1.0,
+    "original_max_position_embeddings": 8192,
+    "rope_type": "llama3"
+  },
+  "rope_theta": 500000.0,
+  "router_aux_loss_coef": 0.001,
+  "router_jitter_noise": 0.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.51.1",
+  "use_cache": true,
+  "use_qk_norm": true,
+  "vocab_size": 202048
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "bos_token_id": 200000,
+  "do_sample": true,
+  "eos_token_id": [
+    200001,
+    200007,
+    200008
+  ],
+  "pad_token_id": 200018,
+  "temperature": 0.6,
+  "top_p": 0.9,
+  "transformers_version": "4.51.1"
+}

model-00001-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b44517196e1cc7aad68a9fed29cf244536af055faeb910485297b5f034a492aa
+size 4879155936

model-00002-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5687511ac909e6d6db4cad27e6707d623ca886231c089418fb04eff9d2d4b7d6
+size 4404205040

model-00003-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:665f2e3f645a5a029c88438f003f78cf2897fc964ed261c5a5a84311a1473e57
+size 4404205040

model-00004-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d7965d2ae00ca62ef5a12113c0cdceeb7c7ca71d435a54cae02085cffffa61a9
+size 4404205040

model-00005-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e7ab522ec7735ca46fd0b67dceeb6641264d6d4809f6b80d78a624ee71a1407
+size 4404205040

model-00006-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c265ae85a8bf9dbcef10ff100c0ad5594e51f0c9cbaa4cf6d4d639a4da3ad5c4
+size 4404205040

model-00007-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49fad02ed80c2340971a8b9865c1f079f1631b8a6faa1b5cd5b5c021f62c271b
+size 4404205040

model-00008-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ed3ffded0e0fb3ee0325cb87015b686749c46717da45682f1ef424e71f0da4b
+size 4404205040

model-00009-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:699ccdc982d2e1a48ecfb5819c3b2dae341b82192a86d39252f6773ad7a91506
+size 4404205040

model-00010-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7d127c2bc7dff9fadd45f45a034a70d989d16be3038672318ed671b608baf82c
+size 4404205040

model-00011-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c4d991c556680ebddf078129416ae9ccb3649754420e293f015f6944e519daa
+size 4404205040

model-00012-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:66888b737cb154904dec902e2547796c0885d8ca2c0e2a9697fbe2f384daba00
+size 4404205048

model-00013-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e2a0902bbd84309ee42b9acc800ac4091e13b625a675672059ee902058cf5ac9
+size 4404205048

model-00014-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd2057ee0c610c68050bc23f542d2ba1abc98575eebe85047396e8ca8ec1db01
+size 4404205048

model-00015-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:129d6b6b1b6344fe8c8b3acb806bf9e0c5ee9a8caee1eeeb70dccd1d02743f02
+size 4404205048

model-00016-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc88efa2cdbcd3c8153bea15fadc55e7f304e4266a4a10089ccf466e11ef26d8
+size 4404205048

model-00017-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c0d6780eaba25011fd6c579939b2c5a0ab0c82328c7c430beb2db87f8f80b41a
+size 4404205048

model-00018-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b9bc30336f1188114520d0ab9637b434f417c196aae1e8c46d2bc0cfb0c93b7
+size 4404205048

model-00019-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:11e88e09c4066accfcdec90def01a4c95743c1637e47ffe81344107ad1c25aa9
+size 4404205048

model-00020-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:798a3557f58ccbb72e5951e53f877b6115dfe913d3f7a7a228c2bf57c5925c99
+size 4404205048

model-00021-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:30294cc6ed865deb82cc4f6249d532c12429d0e018b5434b2db1b25d44a7ee95
+size 4404205048

model-00022-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e527f77286b4e65ea4ebd19bf116a7299f691aaea17d9d86be09a9ad2719ab75
+size 4404205048

model-00023-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d4f0aac0eac9c75b9378cabcaaf32ad08639648d1ac472e2ce75c473025323d6
+size 4404205048

model-00024-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31ba15bd94eab3fdd8f861ce2397c551f96d5587c1ef58999a896e5cefb7a8c5
+size 4404205048

model-00025-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:25324211544eaf72745d6c9bedf69de2a2d298f2ffea728a632a6ca1daa61df0
+size 4404205048

model-00026-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98f01db6e6e786ca5c526d5f8ad0a036650273ff254f9c839525b1262b207739
+size 4404205048

model-00027-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1b2c2489cb33a39481db473765d151a02d1f1394fe42fdf239e67cb98fba4e4
+size 4404205048

model-00028-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:56a0ab930a8cb9bb173ef164c04b2da6c3d01bd250abb93f193454fc68f94b74
+size 4404205048

model-00029-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0b0b105c48ebd9e5acb08e2d2f4aa7e54b3a207e2d64948af406c4c1d1758e59
+size 4404205048

model-00030-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b691698dce078e381cb49ae4ed4482d3dba3a263e698be06cd91816ad696422
+size 4404205048

model-00031-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e6e2a980b450fd29a4a7986b94a04dccfaacd59f45ba95218fcbd262c7bc7fe
+size 4404205048

model-00032-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ef7e4c5abe8f85fdd856ba38ab30c798c4237f1f3d384825e086d84d080fe69
+size 4404205048

model-00033-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:85589d62191034f854077e7dcb1f7e6f223f5c75a6bd7e19abf0479a8e2743ad
+size 4404205048

model-00034-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:44910083a3c6823a456ef3b7a50ecadc9240a75235db62e59f17e0caa2b3a5ac
+size 4404205048

model-00035-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6a0ea0ec4f7ea14fcfe5d11f022b74fed76dc2e232b866f0346c8f1d7c58daa9
+size 4404205048

model-00036-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3bf5338c3f1b4dcc4ea09ae3c04d1dcd0f979d20de38ad16e8e688e719d342f5
+size 4404205048

model-00037-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8c1dc1d69042728cc7a4f64f3c0e9dfdd00713cbf459561ec5908bca396ec0de
+size 4404205048

model-00038-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3204b3596fec5a7a7e88a4348bff922f563d1690b37dfb4acb211b1c3885aba8
+size 4404205048

model-00039-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d48c6ec25f48c2f65ede66ca830d0dd18582298198e085e9b30912137a4794f8
+size 4404205048

model-00040-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0dd5705775999f38b15bb4b4b3ead7f3f4f5aaac9d3b73234515d1b2d2a39f9e
+size 4404205048

model-00041-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3557d952e5e8962f662b100eb8e81891198be2bb4dad8f60b6cb7ec00339d5a6
+size 4404205048

model-00042-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:834f4c0a0f59416b69ce8bd43bf6bd71c8062450ce8dc12f8add375f64549929
+size 4404205048

model-00043-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ae18c698991038f8a466eba19426e0589ce61274f43869580db7bd4452d4aa1
+size 4404205048

model-00044-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b27ec00433760f3cd655cb9d7eb97e57de7b527f39a99789f88a89c9b2732c03
+size 4404205048

model-00045-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4937795dd1f389fb8f08509a571f37b44f33da4b320acce5f7e5da53485276e1
+size 4404205048

model-00046-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:907d51bf74aec90ac1247a83573e68eec916a03325270454a2c5d26b976957b7
+size 4404205048

model-00047-of-00049.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:290687e729c253f146eefa800206dca72637fd83459e53217c7f2342a6a3e233
+size 4404205048