Training in progress, epoch 0, checkpoint

last-checkpoint/README.md

@@ -0,0 +1,202 @@
+---
+base_model: Vikhrmodels/Vikhr-7B-instruct_0.4
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

last-checkpoint/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Vikhrmodels/Vikhr-7B-instruct_0.4",
+  "bias": "none",
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.15,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "q_proj",
+    "gate_proj",
+    "down_proj",
+    "v_proj",
+    "o_proj",
+    "up_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

last-checkpoint/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:856300bfe73a830e55fa3fa014fdcad332aa948b01752816a36770d3ee567a0a
+size 335604696

last-checkpoint/added_tokens.json

@@ -0,0 +1,16 @@
+{
+  "<|future_token_0|>": 79085,
+  "<|future_token_10|>": 79095,
+  "<|future_token_11|>": 79096,
+  "<|future_token_1|>": 79086,
+  "<|future_token_2|>": 79087,
+  "<|future_token_3|>": 79088,
+  "<|future_token_4|>": 79089,
+  "<|future_token_5|>": 79090,
+  "<|future_token_6|>": 79091,
+  "<|future_token_7|>": 79092,
+  "<|future_token_8|>": 79093,
+  "<|future_token_9|>": 79094,
+  "<|im_end|>": 79097,
+  "<|im_start|>": 79098
+}

last-checkpoint/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57f15e8aeb7b97e4aeb20e8f4fea06bf2b2f9c793bb543ac55b481d9d1a1505f
+size 671473298

last-checkpoint/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5662b638b4df2e7c84af03aca5fec1504c27e9a0fe1663035c63907d33554253
+size 14244

last-checkpoint/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:891cad020bf7bee78efa739dc10e1e4315e34b096ed70226b38590ec81d7d418
+size 1064

last-checkpoint/special_tokens_map.json

@@ -0,0 +1,34 @@
+{
+  "additional_special_tokens": [
+    "<|im_end|>",
+    "<|im_start|>"
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

last-checkpoint/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3f0f84b42f79947ab92931c7c5ded699e845e6862e4e29f0b97f87e279da601
+size 1636166

last-checkpoint/tokenizer_config.json

@@ -0,0 +1,159 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "79085": {
+      "content": "<|future_token_0|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79086": {
+      "content": "<|future_token_1|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79087": {
+      "content": "<|future_token_2|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79088": {
+      "content": "<|future_token_3|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79089": {
+      "content": "<|future_token_4|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79090": {
+      "content": "<|future_token_5|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79091": {
+      "content": "<|future_token_6|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79092": {
+      "content": "<|future_token_7|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79093": {
+      "content": "<|future_token_8|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79094": {
+      "content": "<|future_token_9|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79095": {
+      "content": "<|future_token_10|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79096": {
+      "content": "<|future_token_11|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79097": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "79098": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_end|>",
+    "<|im_start|>"
+  ],
+  "bos_token": "<s>",
+  "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": true,
+  "model_max_length": 1512,
+  "pad_token": "</s>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": true
+}

last-checkpoint/trainer_state.json

@@ -0,0 +1,1531 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.03932698421021584,
+  "eval_steps": 500,
+  "global_step": 1500,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.00018352592631434058,
+      "grad_norm": 6.47861909866333,
+      "learning_rate": 7.000000000000001e-06,
+      "loss": 1.3763,
+      "step": 7
+    },
+    {
+      "epoch": 0.00036705185262868117,
+      "grad_norm": 2.5405163764953613,
+      "learning_rate": 1.4000000000000001e-05,
+      "loss": 0.2669,
+      "step": 14
+    },
+    {
+      "epoch": 0.0005505777789430217,
+      "grad_norm": 0.17570653557777405,
+      "learning_rate": 2.1e-05,
+      "loss": 0.0553,
+      "step": 21
+    },
+    {
+      "epoch": 0.0007341037052573623,
+      "grad_norm": 0.13182853162288666,
+      "learning_rate": 2.8000000000000003e-05,
+      "loss": 0.0754,
+      "step": 28
+    },
+    {
+      "epoch": 0.0009176296315717029,
+      "grad_norm": 1.6620056629180908,
+      "learning_rate": 3.5e-05,
+      "loss": 0.0674,
+      "step": 35
+    },
+    {
+      "epoch": 0.0011011555578860435,
+      "grad_norm": 7.971460342407227,
+      "learning_rate": 4.2e-05,
+      "loss": 0.0592,
+      "step": 42
+    },
+    {
+      "epoch": 0.0012846814842003842,
+      "grad_norm": 0.45227354764938354,
+      "learning_rate": 4.9e-05,
+      "loss": 0.0373,
+      "step": 49
+    },
+    {
+      "epoch": 0.0014682074105147247,
+      "grad_norm": 0.3094736635684967,
+      "learning_rate": 5.6000000000000006e-05,
+      "loss": 0.0274,
+      "step": 56
+    },
+    {
+      "epoch": 0.0016517333368290654,
+      "grad_norm": 6.417137622833252,
+      "learning_rate": 6.3e-05,
+      "loss": 0.0403,
+      "step": 63
+    },
+    {
+      "epoch": 0.0018352592631434058,
+      "grad_norm": 15.522113800048828,
+      "learning_rate": 7e-05,
+      "loss": 0.1099,
+      "step": 70
+    },
+    {
+      "epoch": 0.0020187851894577463,
+      "grad_norm": 7.321249961853027,
+      "learning_rate": 7.7e-05,
+      "loss": 0.1413,
+      "step": 77
+    },
+    {
+      "epoch": 0.002202311115772087,
+      "grad_norm": 8.105692863464355,
+      "learning_rate": 8.4e-05,
+      "loss": 0.083,
+      "step": 84
+    },
+    {
+      "epoch": 0.0023858370420864277,
+      "grad_norm": 2.9659645557403564,
+      "learning_rate": 9.1e-05,
+      "loss": 0.1051,
+      "step": 91
+    },
+    {
+      "epoch": 0.0025693629684007684,
+      "grad_norm": 59.94154357910156,
+      "learning_rate": 9.8e-05,
+      "loss": 0.0305,
+      "step": 98
+    },
+    {
+      "epoch": 0.0027528888947151086,
+      "grad_norm": 2.124544143676758,
+      "learning_rate": 9.999685283773504e-05,
+      "loss": 0.065,
+      "step": 105
+    },
+    {
+      "epoch": 0.0029364148210294493,
+      "grad_norm": 2.3349404335021973,
+      "learning_rate": 9.998187325055106e-05,
+      "loss": 0.124,
+      "step": 112
+    },
+    {
+      "epoch": 0.00311994074734379,
+      "grad_norm": 0.6199296116828918,
+      "learning_rate": 9.995456138403733e-05,
+      "loss": 0.0371,
+      "step": 119
+    },
+    {
+      "epoch": 0.0033034666736581307,
+      "grad_norm": 39.53347396850586,
+      "learning_rate": 9.991492397698826e-05,
+      "loss": 0.1866,
+      "step": 126
+    },
+    {
+      "epoch": 0.003486992599972471,
+      "grad_norm": 25.955495834350586,
+      "learning_rate": 9.986297080934089e-05,
+      "loss": 0.1097,
+      "step": 133
+    },
+    {
+      "epoch": 0.0036705185262868117,
+      "grad_norm": 1.307746171951294,
+      "learning_rate": 9.979871469976196e-05,
+      "loss": 0.1424,
+      "step": 140
+    },
+    {
+      "epoch": 0.0038540444526011523,
+      "grad_norm": 9.032373428344727,
+      "learning_rate": 9.972217150248503e-05,
+      "loss": 0.2516,
+      "step": 147
+    },
+    {
+      "epoch": 0.004037570378915493,
+      "grad_norm": 10.674254417419434,
+      "learning_rate": 9.963336010339868e-05,
+      "loss": 0.0544,
+      "step": 154
+    },
+    {
+      "epoch": 0.004221096305229833,
+      "grad_norm": 4.597137451171875,
+      "learning_rate": 9.953230241538674e-05,
+      "loss": 0.078,
+      "step": 161
+    },
+    {
+      "epoch": 0.004404622231544174,
+      "grad_norm": 18.2402400970459,
+      "learning_rate": 9.941902337292155e-05,
+      "loss": 0.1367,
+      "step": 168
+    },
+    {
+      "epoch": 0.004588148157858515,
+      "grad_norm": 8.528764724731445,
+      "learning_rate": 9.92935509259118e-05,
+      "loss": 0.106,
+      "step": 175
+    },
+    {
+      "epoch": 0.004771674084172855,
+      "grad_norm": 5.531329154968262,
+      "learning_rate": 9.915591603280631e-05,
+      "loss": 0.0531,
+      "step": 182
+    },
+    {
+      "epoch": 0.004955200010487196,
+      "grad_norm": 10.273998260498047,
+      "learning_rate": 9.900615265295552e-05,
+      "loss": 0.0902,
+      "step": 189
+    },
+    {
+      "epoch": 0.005138725936801537,
+      "grad_norm": 26.211259841918945,
+      "learning_rate": 9.884429773823239e-05,
+      "loss": 0.0418,
+      "step": 196
+    },
+    {
+      "epoch": 0.005322251863115877,
+      "grad_norm": 31.299962997436523,
+      "learning_rate": 9.867039122391527e-05,
+      "loss": 0.2439,
+      "step": 203
+    },
+    {
+      "epoch": 0.005505777789430217,
+      "grad_norm": 6.080491065979004,
+      "learning_rate": 9.848447601883435e-05,
+      "loss": 0.0974,
+      "step": 210
+    },
+    {
+      "epoch": 0.005689303715744558,
+      "grad_norm": 13.236441612243652,
+      "learning_rate": 9.828659799478456e-05,
+      "loss": 0.074,
+      "step": 217
+    },
+    {
+      "epoch": 0.005872829642058899,
+      "grad_norm": 41.40345001220703,
+      "learning_rate": 9.807680597520746e-05,
+      "loss": 0.0806,
+      "step": 224
+    },
+    {
+      "epoch": 0.006056355568373239,
+      "grad_norm": 16.45723533630371,
+      "learning_rate": 9.785515172314463e-05,
+      "loss": 0.1008,
+      "step": 231
+    },
+    {
+      "epoch": 0.00623988149468758,
+      "grad_norm": 0.8506945967674255,
+      "learning_rate": 9.762168992846614e-05,
+      "loss": 0.0881,
+      "step": 238
+    },
+    {
+      "epoch": 0.006423407421001921,
+      "grad_norm": 133.6859588623047,
+      "learning_rate": 9.737647819437645e-05,
+      "loss": 0.2072,
+      "step": 245
+    },
+    {
+      "epoch": 0.006606933347316261,
+      "grad_norm": 0.6294188499450684,
+      "learning_rate": 9.711957702320175e-05,
+      "loss": 0.0809,
+      "step": 252
+    },
+    {
+      "epoch": 0.006790459273630601,
+      "grad_norm": 7.2060160636901855,
+      "learning_rate": 9.685104980146193e-05,
+      "loss": 0.0649,
+      "step": 259
+    },
+    {
+      "epoch": 0.006973985199944942,
+      "grad_norm": 13.655864715576172,
+      "learning_rate": 9.657096278423093e-05,
+      "loss": 0.1062,
+      "step": 266
+    },
+    {
+      "epoch": 0.007157511126259283,
+      "grad_norm": 214.4697265625,
+      "learning_rate": 9.627938507878917e-05,
+      "loss": 0.0951,
+      "step": 273
+    },
+    {
+      "epoch": 0.007341037052573623,
+      "grad_norm": 3.5746772289276123,
+      "learning_rate": 9.597638862757255e-05,
+      "loss": 0.0428,
+      "step": 280
+    },
+    {
+      "epoch": 0.007524562978887964,
+      "grad_norm": 4.098369121551514,
+      "learning_rate": 9.566204819042152e-05,
+      "loss": 0.1565,
+      "step": 287
+    },
+    {
+      "epoch": 0.007708088905202305,
+      "grad_norm": 7.281524181365967,
+      "learning_rate": 9.533644132613541e-05,
+      "loss": 0.0639,
+      "step": 294
+    },
+    {
+      "epoch": 0.007891614831516645,
+      "grad_norm": 6.041608810424805,
+      "learning_rate": 9.49996483733358e-05,
+      "loss": 0.0507,
+      "step": 301
+    },
+    {
+      "epoch": 0.008075140757830985,
+      "grad_norm": 16.09194564819336,
+      "learning_rate": 9.465175243064428e-05,
+      "loss": 0.0462,
+      "step": 308
+    },
+    {
+      "epoch": 0.008258666684145327,
+      "grad_norm": 3.559749126434326,
+      "learning_rate": 9.4292839336179e-05,
+      "loss": 0.1424,
+      "step": 315
+    },
+    {
+      "epoch": 0.008442192610459667,
+      "grad_norm": 9.500808715820312,
+      "learning_rate": 9.39229976463755e-05,
+      "loss": 0.0839,
+      "step": 322
+    },
+    {
+      "epoch": 0.008625718536774008,
+      "grad_norm": 3.6868910789489746,
+      "learning_rate": 9.354231861413668e-05,
+      "loss": 0.0591,
+      "step": 329
+    },
+    {
+      "epoch": 0.008809244463088348,
+      "grad_norm": 3.754107713699341,
+      "learning_rate": 9.315089616631752e-05,
+      "loss": 0.0855,
+      "step": 336
+    },
+    {
+      "epoch": 0.008992770389402688,
+      "grad_norm": 13.147061347961426,
+      "learning_rate": 9.274882688055005e-05,
+      "loss": 0.3253,
+      "step": 343
+    },
+    {
+      "epoch": 0.00917629631571703,
+      "grad_norm": 12.47993278503418,
+      "learning_rate": 9.233620996141421e-05,
+      "loss": 0.096,
+      "step": 350
+    },
+    {
+      "epoch": 0.00935982224203137,
+      "grad_norm": 7.317345142364502,
+      "learning_rate": 9.191314721596072e-05,
+      "loss": 0.0446,
+      "step": 357
+    },
+    {
+      "epoch": 0.00954334816834571,
+      "grad_norm": 17.10198211669922,
+      "learning_rate": 9.147974302859157e-05,
+      "loss": 0.112,
+      "step": 364
+    },
+    {
+      "epoch": 0.00972687409466005,
+      "grad_norm": 27.49844741821289,
+      "learning_rate": 9.103610433530483e-05,
+      "loss": 0.1984,
+      "step": 371
+    },
+    {
+      "epoch": 0.009910400020974392,
+      "grad_norm": 10.77637767791748,
+      "learning_rate": 9.058234059730976e-05,
+      "loss": 0.0612,
+      "step": 378
+    },
+    {
+      "epoch": 0.010093925947288732,
+      "grad_norm": 8.21988582611084,
+      "learning_rate": 9.01185637740189e-05,
+      "loss": 0.1497,
+      "step": 385
+    },
+    {
+      "epoch": 0.010277451873603074,
+      "grad_norm": 11.31648063659668,
+      "learning_rate": 8.964488829542377e-05,
+      "loss": 0.0579,
+      "step": 392
+    },
+    {
+      "epoch": 0.010460977799917413,
+      "grad_norm": 18.27166175842285,
+      "learning_rate": 8.916143103386093e-05,
+      "loss": 0.1129,
+      "step": 399
+    },
+    {
+      "epoch": 0.010644503726231753,
+      "grad_norm": 9.18415641784668,
+      "learning_rate": 8.866831127517557e-05,
+      "loss": 0.0659,
+      "step": 406
+    },
+    {
+      "epoch": 0.010828029652546095,
+      "grad_norm": 3.2998199462890625,
+      "learning_rate": 8.81656506892894e-05,
+      "loss": 0.0525,
+      "step": 413
+    },
+    {
+      "epoch": 0.011011555578860435,
+      "grad_norm": 6.025943279266357,
+      "learning_rate": 8.765357330018056e-05,
+      "loss": 0.04,
+      "step": 420
+    },
+    {
+      "epoch": 0.011195081505174776,
+      "grad_norm": 9.952337265014648,
+      "learning_rate": 8.71322054552824e-05,
+      "loss": 0.1309,
+      "step": 427
+    },
+    {
+      "epoch": 0.011378607431489116,
+      "grad_norm": 3.3426525592803955,
+      "learning_rate": 8.660167579430927e-05,
+      "loss": 0.108,
+      "step": 434
+    },
+    {
+      "epoch": 0.011562133357803457,
+      "grad_norm": 8.256426811218262,
+      "learning_rate": 8.606211521751652e-05,
+      "loss": 0.1007,
+      "step": 441
+    },
+    {
+      "epoch": 0.011745659284117797,
+      "grad_norm": 17.211164474487305,
+      "learning_rate": 8.551365685340285e-05,
+      "loss": 0.0535,
+      "step": 448
+    },
+    {
+      "epoch": 0.011929185210432139,
+      "grad_norm": 97.90534210205078,
+      "learning_rate": 8.495643602586287e-05,
+      "loss": 0.3113,
+      "step": 455
+    },
+    {
+      "epoch": 0.012112711136746479,
+      "grad_norm": 2.0993361473083496,
+      "learning_rate": 8.439059022079789e-05,
+      "loss": 0.0717,
+      "step": 462
+    },
+    {
+      "epoch": 0.012296237063060819,
+      "grad_norm": 7.462172031402588,
+      "learning_rate": 8.381625905219339e-05,
+      "loss": 0.0334,
+      "step": 469
+    },
+    {
+      "epoch": 0.01247976298937516,
+      "grad_norm": 3.631150722503662,
+      "learning_rate": 8.32335842276713e-05,
+      "loss": 0.0554,
+      "step": 476
+    },
+    {
+      "epoch": 0.0126632889156895,
+      "grad_norm": 38.864410400390625,
+      "learning_rate": 8.264270951352581e-05,
+      "loss": 0.0669,
+      "step": 483
+    },
+    {
+      "epoch": 0.012846814842003841,
+      "grad_norm": 19.04292106628418,
+      "learning_rate": 8.20437806992512e-05,
+      "loss": 0.0942,
+      "step": 490
+    },
+    {
+      "epoch": 0.013030340768318181,
+      "grad_norm": 7.4071784019470215,
+      "learning_rate": 8.143694556157046e-05,
+      "loss": 0.0376,
+      "step": 497
+    },
+    {
+      "epoch": 0.013213866694632523,
+      "grad_norm": 2.958404064178467,
+      "learning_rate": 8.082235382797349e-05,
+      "loss": 0.0295,
+      "step": 504
+    },
+    {
+      "epoch": 0.013397392620946863,
+      "grad_norm": 11.70897388458252,
+      "learning_rate": 8.020015713977427e-05,
+      "loss": 0.0745,
+      "step": 511
+    },
+    {
+      "epoch": 0.013580918547261202,
+      "grad_norm": 37.44210433959961,
+      "learning_rate": 7.957050901469545e-05,
+      "loss": 0.121,
+      "step": 518
+    },
+    {
+      "epoch": 0.013764444473575544,
+      "grad_norm": 15.786369323730469,
+      "learning_rate": 7.89335648089903e-05,
+      "loss": 0.0618,
+      "step": 525
+    },
+    {
+      "epoch": 0.013947970399889884,
+      "grad_norm": 9.872979164123535,
+      "learning_rate": 7.828948167911074e-05,
+      "loss": 0.0326,
+      "step": 532
+    },
+    {
+      "epoch": 0.014131496326204225,
+      "grad_norm": 22.587263107299805,
+      "learning_rate": 7.763841854293145e-05,
+      "loss": 0.0565,
+      "step": 539
+    },
+    {
+      "epoch": 0.014315022252518565,
+      "grad_norm": 1.4639612436294556,
+      "learning_rate": 7.698053604053922e-05,
+      "loss": 0.0832,
+      "step": 546
+    },
+    {
+      "epoch": 0.014498548178832907,
+      "grad_norm": 25.581790924072266,
+      "learning_rate": 7.631599649459744e-05,
+      "loss": 0.1151,
+      "step": 553
+    },
+    {
+      "epoch": 0.014682074105147247,
+      "grad_norm": 0.8175346851348877,
+      "learning_rate": 7.564496387029532e-05,
+      "loss": 0.0409,
+      "step": 560
+    },
+    {
+      "epoch": 0.014865600031461588,
+      "grad_norm": 0.31356096267700195,
+      "learning_rate": 7.496760373489202e-05,
+      "loss": 0.0361,
+      "step": 567
+    },
+    {
+      "epoch": 0.015049125957775928,
+      "grad_norm": 12.69936466217041,
+      "learning_rate": 7.428408321686541e-05,
+      "loss": 0.0961,
+      "step": 574
+    },
+    {
+      "epoch": 0.015232651884090268,
+      "grad_norm": 23.00505828857422,
+      "learning_rate": 7.35945709646756e-05,
+      "loss": 0.041,
+      "step": 581
+    },
+    {
+      "epoch": 0.01541617781040461,
+      "grad_norm": 3.063689708709717,
+      "learning_rate": 7.289923710515339e-05,
+      "loss": 0.0747,
+      "step": 588
+    },
+    {
+      "epoch": 0.01559970373671895,
+      "grad_norm": 9.534968376159668,
+      "learning_rate": 7.219825320152411e-05,
+      "loss": 0.0514,
+      "step": 595
+    },
+    {
+      "epoch": 0.01578322966303329,
+      "grad_norm": 12.815621376037598,
+      "learning_rate": 7.149179221107694e-05,
+      "loss": 0.1077,
+      "step": 602
+    },
+    {
+      "epoch": 0.015966755589347632,
+      "grad_norm": 14.98495864868164,
+      "learning_rate": 7.078002844249032e-05,
+      "loss": 0.0369,
+      "step": 609
+    },
+    {
+      "epoch": 0.01615028151566197,
+      "grad_norm": 4.274013519287109,
+      "learning_rate": 7.006313751282372e-05,
+      "loss": 0.0288,
+      "step": 616
+    },
+    {
+      "epoch": 0.016333807441976312,
+      "grad_norm": 11.138263702392578,
+      "learning_rate": 6.934129630418701e-05,
+      "loss": 0.0524,
+      "step": 623
+    },
+    {
+      "epoch": 0.016517333368290654,
+      "grad_norm": 2.7880783081054688,
+      "learning_rate": 6.861468292009727e-05,
+      "loss": 0.1132,
+      "step": 630
+    },
+    {
+      "epoch": 0.01670085929460499,
+      "grad_norm": 6.396130561828613,
+      "learning_rate": 6.788347664153447e-05,
+      "loss": 0.1058,
+      "step": 637
+    },
+    {
+      "epoch": 0.016884385220919333,
+      "grad_norm": 8.992058753967285,
+      "learning_rate": 6.714785788270658e-05,
+      "loss": 0.153,
+      "step": 644
+    },
+    {
+      "epoch": 0.017067911147233675,
+      "grad_norm": 3.507546901702881,
+      "learning_rate": 6.640800814653503e-05,
+      "loss": 0.0426,
+      "step": 651
+    },
+    {
+      "epoch": 0.017251437073548016,
+      "grad_norm": 18.117786407470703,
+      "learning_rate": 6.566410997987163e-05,
+      "loss": 0.4948,
+      "step": 658
+    },
+    {
+      "epoch": 0.017434962999862354,
+      "grad_norm": 13.964621543884277,
+      "learning_rate": 6.49163469284578e-05,
+      "loss": 0.0528,
+      "step": 665
+    },
+    {
+      "epoch": 0.017618488926176696,
+      "grad_norm": 3.142885446548462,
+      "learning_rate": 6.416490349163748e-05,
+      "loss": 0.0466,
+      "step": 672
+    },
+    {
+      "epoch": 0.017802014852491038,
+      "grad_norm": 7.793853282928467,
+      "learning_rate": 6.340996507683458e-05,
+      "loss": 0.0259,
+      "step": 679
+    },
+    {
+      "epoch": 0.017985540778805376,
+      "grad_norm": 18.73122787475586,
+      "learning_rate": 6.265171795380659e-05,
+      "loss": 0.0687,
+      "step": 686
+    },
+    {
+      "epoch": 0.018169066705119717,
+      "grad_norm": 72.39578247070312,
+      "learning_rate": 6.189034920868522e-05,
+      "loss": 0.0404,
+      "step": 693
+    },
+    {
+      "epoch": 0.01835259263143406,
+      "grad_norm": 12.056116104125977,
+      "learning_rate": 6.112604669781572e-05,
+      "loss": 0.0681,
+      "step": 700
+    },
+    {
+      "epoch": 0.0185361185577484,
+      "grad_norm": 42.570858001708984,
+      "learning_rate": 6.0358999001406156e-05,
+      "loss": 0.0402,
+      "step": 707
+    },
+    {
+      "epoch": 0.01871964448406274,
+      "grad_norm": 7.0034308433532715,
+      "learning_rate": 5.9589395376998e-05,
+      "loss": 0.047,
+      "step": 714
+    },
+    {
+      "epoch": 0.01890317041037708,
+      "grad_norm": 24.041772842407227,
+      "learning_rate": 5.8817425712769794e-05,
+      "loss": 0.0346,
+      "step": 721
+    },
+    {
+      "epoch": 0.01908669633669142,
+      "grad_norm": 11.524798393249512,
+      "learning_rate": 5.804328048068492e-05,
+      "loss": 0.0779,
+      "step": 728
+    },
+    {
+      "epoch": 0.019270222263005763,
+      "grad_norm": 12.005279541015625,
+      "learning_rate": 5.7267150689495644e-05,
+      "loss": 0.1136,
+      "step": 735
+    },
+    {
+      "epoch": 0.0194537481893201,
+      "grad_norm": 11.910235404968262,
+      "learning_rate": 5.648922783761443e-05,
+      "loss": 0.0734,
+      "step": 742
+    },
+    {
+      "epoch": 0.019637274115634443,
+      "grad_norm": 10.383480072021484,
+      "learning_rate": 5.570970386586469e-05,
+      "loss": 0.0665,
+      "step": 749
+    },
+    {
+      "epoch": 0.019820800041948784,
+      "grad_norm": 30.089757919311523,
+      "learning_rate": 5.492877111012218e-05,
+      "loss": 0.1305,
+      "step": 756
+    },
+    {
+      "epoch": 0.020004325968263122,
+      "grad_norm": 14.836136817932129,
+      "learning_rate": 5.414662225385903e-05,
+      "loss": 0.0719,
+      "step": 763
+    },
+    {
+      "epoch": 0.020187851894577464,
+      "grad_norm": 4.434623718261719,
+      "learning_rate": 5.336345028060199e-05,
+      "loss": 0.0243,
+      "step": 770
+    },
+    {
+      "epoch": 0.020371377820891805,
+      "grad_norm": 10.032936096191406,
+      "learning_rate": 5.257944842631658e-05,
+      "loss": 0.0509,
+      "step": 777
+    },
+    {
+      "epoch": 0.020554903747206147,
+      "grad_norm": 12.639185905456543,
+      "learning_rate": 5.179481013172912e-05,
+      "loss": 0.0605,
+      "step": 784
+    },
+    {
+      "epoch": 0.020738429673520485,
+      "grad_norm": 23.700876235961914,
+      "learning_rate": 5.100972899459796e-05,
+      "loss": 0.0219,
+      "step": 791
+    },
+    {
+      "epoch": 0.020921955599834827,
+      "grad_norm": 5.223752498626709,
+      "learning_rate": 5.022439872194629e-05,
+      "loss": 0.0434,
+      "step": 798
+    },
+    {
+      "epoch": 0.021105481526149168,
+      "grad_norm": 2.694279432296753,
+      "learning_rate": 4.943901308226771e-05,
+      "loss": 0.0399,
+      "step": 805
+    },
+    {
+      "epoch": 0.021289007452463506,
+      "grad_norm": 0.11702734231948853,
+      "learning_rate": 4.865376585771687e-05,
+      "loss": 0.0231,
+      "step": 812
+    },
+    {
+      "epoch": 0.021472533378777848,
+      "grad_norm": 4.123202323913574,
+      "learning_rate": 4.7868850796296495e-05,
+      "loss": 0.0467,
+      "step": 819
+    },
+    {
+      "epoch": 0.02165605930509219,
+      "grad_norm": 44.92766571044922,
+      "learning_rate": 4.708446156405307e-05,
+      "loss": 0.0715,
+      "step": 826
+    },
+    {
+      "epoch": 0.02183958523140653,
+      "grad_norm": 0.7099213004112244,
+      "learning_rate": 4.630079169729257e-05,
+      "loss": 0.0173,
+      "step": 833
+    },
+    {
+      "epoch": 0.02202311115772087,
+      "grad_norm": 3.2569453716278076,
+      "learning_rate": 4.551803455482833e-05,
+      "loss": 0.029,
+      "step": 840
+    },
+    {
+      "epoch": 0.02220663708403521,
+      "grad_norm": 2.3254077434539795,
+      "learning_rate": 4.473638327027259e-05,
+      "loss": 0.0173,
+      "step": 847
+    },
+    {
+      "epoch": 0.022390163010349552,
+      "grad_norm": 17.07633399963379,
+      "learning_rate": 4.395603070438373e-05,
+      "loss": 0.0846,
+      "step": 854
+    },
+    {
+      "epoch": 0.02257368893666389,
+      "grad_norm": 198.6379852294922,
+      "learning_rate": 4.31771693974807e-05,
+      "loss": 0.0813,
+      "step": 861
+    },
+    {
+      "epoch": 0.022757214862978232,
+      "grad_norm": 0.7852660417556763,
+      "learning_rate": 4.239999152193664e-05,
+      "loss": 0.0355,
+      "step": 868
+    },
+    {
+      "epoch": 0.022940740789292573,
+      "grad_norm": 2.008527994155884,
+      "learning_rate": 4.162468883476319e-05,
+      "loss": 0.0683,
+      "step": 875
+    },
+    {
+      "epoch": 0.023124266715606915,
+      "grad_norm": 205.48631286621094,
+      "learning_rate": 4.085145263029726e-05,
+      "loss": 0.2176,
+      "step": 882
+    },
+    {
+      "epoch": 0.023307792641921253,
+      "grad_norm": 267.4538269042969,
+      "learning_rate": 4.008047369300218e-05,
+      "loss": 0.4272,
+      "step": 889
+    },
+    {
+      "epoch": 0.023491318568235595,
+      "grad_norm": 6.071589469909668,
+      "learning_rate": 3.9311942250394276e-05,
+      "loss": 0.1293,
+      "step": 896
+    },
+    {
+      "epoch": 0.023674844494549936,
+      "grad_norm": 9.108537673950195,
+      "learning_rate": 3.8546047926107256e-05,
+      "loss": 0.0102,
+      "step": 903
+    },
+    {
+      "epoch": 0.023858370420864278,
+      "grad_norm": 13.065569877624512,
+      "learning_rate": 3.778297969310529e-05,
+      "loss": 0.0246,
+      "step": 910
+    },
+    {
+      "epoch": 0.024041896347178616,
+      "grad_norm": 6.456232070922852,
+      "learning_rate": 3.7022925827056884e-05,
+      "loss": 0.0451,
+      "step": 917
+    },
+    {
+      "epoch": 0.024225422273492957,
+      "grad_norm": 16.841550827026367,
+      "learning_rate": 3.62660738598805e-05,
+      "loss": 0.0338,
+      "step": 924
+    },
+    {
+      "epoch": 0.0244089481998073,
+      "grad_norm": 0.009042863734066486,
+      "learning_rate": 3.551261053347404e-05,
+      "loss": 0.0103,
+      "step": 931
+    },
+    {
+      "epoch": 0.024592474126121637,
+      "grad_norm": 9.744179725646973,
+      "learning_rate": 3.4762721753638995e-05,
+      "loss": 0.0386,
+      "step": 938
+    },
+    {
+      "epoch": 0.02477600005243598,
+      "grad_norm": 6.837308406829834,
+      "learning_rate": 3.401659254421094e-05,
+      "loss": 0.0391,
+      "step": 945
+    },
+    {
+      "epoch": 0.02495952597875032,
+      "grad_norm": 10.45736026763916,
+      "learning_rate": 3.3274407001407735e-05,
+      "loss": 0.0213,
+      "step": 952
+    },
+    {
+      "epoch": 0.02514305190506466,
+      "grad_norm": 16.186050415039062,
+      "learning_rate": 3.2536348248406534e-05,
+      "loss": 0.0168,
+      "step": 959
+    },
+    {
+      "epoch": 0.025326577831379,
+      "grad_norm": 11.363747596740723,
+      "learning_rate": 3.1802598390160784e-05,
+      "loss": 0.096,
+      "step": 966
+    },
+    {
+      "epoch": 0.02551010375769334,
+      "grad_norm": 2.184530019760132,
+      "learning_rate": 3.107333846846872e-05,
+      "loss": 0.0133,
+      "step": 973
+    },
+    {
+      "epoch": 0.025693629684007683,
+      "grad_norm": 4.363285541534424,
+      "learning_rate": 3.0348748417303823e-05,
+      "loss": 0.0137,
+      "step": 980
+    },
+    {
+      "epoch": 0.02587715561032202,
+      "grad_norm": 0.1319400668144226,
+      "learning_rate": 2.9629007018418985e-05,
+      "loss": 0.0025,
+      "step": 987
+    },
+    {
+      "epoch": 0.026060681536636363,
+      "grad_norm": 2.4574859142303467,
+      "learning_rate": 2.8914291857234636e-05,
+      "loss": 0.0275,
+      "step": 994
+    },
+    {
+      "epoch": 0.026244207462950704,
+      "grad_norm": 1.7651581764221191,
+      "learning_rate": 2.8204779279022276e-05,
+      "loss": 0.0049,
+      "step": 1001
+    },
+    {
+      "epoch": 0.026427733389265046,
+      "grad_norm": 34.096675872802734,
+      "learning_rate": 2.7500644345393943e-05,
+      "loss": 0.0254,
+      "step": 1008
+    },
+    {
+      "epoch": 0.026611259315579384,
+      "grad_norm": 9.816268920898438,
+      "learning_rate": 2.68020607911083e-05,
+      "loss": 0.0148,
+      "step": 1015
+    },
+    {
+      "epoch": 0.026794785241893725,
+      "grad_norm": 4.333038330078125,
+      "learning_rate": 2.610920098120424e-05,
+      "loss": 0.0126,
+      "step": 1022
+    },
+    {
+      "epoch": 0.026978311168208067,
+      "grad_norm": 0.14241164922714233,
+      "learning_rate": 2.5422235868472345e-05,
+      "loss": 0.0018,
+      "step": 1029
+    },
+    {
+      "epoch": 0.027161837094522405,
+      "grad_norm": 0.08830176293849945,
+      "learning_rate": 2.4741334951274947e-05,
+      "loss": 0.0304,
+      "step": 1036
+    },
+    {
+      "epoch": 0.027345363020836747,
+      "grad_norm": 0.0071082874201238155,
+      "learning_rate": 2.40666662317248e-05,
+      "loss": 0.1199,
+      "step": 1043
+    },
+    {
+      "epoch": 0.027528888947151088,
+      "grad_norm": 6.611972332000732,
+      "learning_rate": 2.3398396174233178e-05,
+      "loss": 0.0152,
+      "step": 1050
+    },
+    {
+      "epoch": 0.02771241487346543,
+      "grad_norm": 0.012415258213877678,
+      "learning_rate": 2.2736689664437217e-05,
+      "loss": 0.008,
+      "step": 1057
+    },
+    {
+      "epoch": 0.027895940799779768,
+      "grad_norm": 0.032711971551179886,
+      "learning_rate": 2.2081709968516866e-05,
+      "loss": 0.0498,
+      "step": 1064
+    },
+    {
+      "epoch": 0.02807946672609411,
+      "grad_norm": 0.18527355790138245,
+      "learning_rate": 2.1433618692911467e-05,
+      "loss": 0.0391,
+      "step": 1071
+    },
+    {
+      "epoch": 0.02826299265240845,
+      "grad_norm": 6.831684589385986,
+      "learning_rate": 2.0792575744445653e-05,
+      "loss": 0.0099,
+      "step": 1078
+    },
+    {
+      "epoch": 0.02844651857872279,
+      "grad_norm": 3.6883180141448975,
+      "learning_rate": 2.015873929087482e-05,
+      "loss": 0.065,
+      "step": 1085
+    },
+    {
+      "epoch": 0.02863004450503713,
+      "grad_norm": 0.03432440012693405,
+      "learning_rate": 1.95322657218596e-05,
+      "loss": 0.0142,
+      "step": 1092
+    },
+    {
+      "epoch": 0.028813570431351472,
+      "grad_norm": 2.314697265625,
+      "learning_rate": 1.8913309610379015e-05,
+      "loss": 0.0416,
+      "step": 1099
+    },
+    {
+      "epoch": 0.028997096357665814,
+      "grad_norm": 8.827579498291016,
+      "learning_rate": 1.8302023674591935e-05,
+      "loss": 0.0268,
+      "step": 1106
+    },
+    {
+      "epoch": 0.02918062228398015,
+      "grad_norm": 0.013051838614046574,
+      "learning_rate": 1.7698558740156135e-05,
+      "loss": 0.0466,
+      "step": 1113
+    },
+    {
+      "epoch": 0.029364148210294493,
+      "grad_norm": 7.55225133895874,
+      "learning_rate": 1.7103063703014372e-05,
+      "loss": 0.0308,
+      "step": 1120
+    },
+    {
+      "epoch": 0.029547674136608835,
+      "grad_norm": 0.3242104947566986,
+      "learning_rate": 1.6515685492656467e-05,
+      "loss": 0.0073,
+      "step": 1127
+    },
+    {
+      "epoch": 0.029731200062923176,
+      "grad_norm": 0.18661095201969147,
+      "learning_rate": 1.59365690358667e-05,
+      "loss": 0.019,
+      "step": 1134
+    },
+    {
+      "epoch": 0.029914725989237514,
+      "grad_norm": 0.19010868668556213,
+      "learning_rate": 1.5365857220965275e-05,
+      "loss": 0.0123,
+      "step": 1141
+    },
+    {
+      "epoch": 0.030098251915551856,
+      "grad_norm": 0.3043920695781708,
+      "learning_rate": 1.4803690862552755e-05,
+      "loss": 0.0058,
+      "step": 1148
+    },
+    {
+      "epoch": 0.030281777841866198,
+      "grad_norm": 0.9419413208961487,
+      "learning_rate": 1.4250208666766235e-05,
+      "loss": 0.0012,
+      "step": 1155
+    },
+    {
+      "epoch": 0.030465303768180536,
+      "grad_norm": 5.666136741638184,
+      "learning_rate": 1.3705547197055584e-05,
+      "loss": 0.0413,
+      "step": 1162
+    },
+    {
+      "epoch": 0.030648829694494877,
+      "grad_norm": 0.7531316876411438,
+      "learning_rate": 1.3169840840488501e-05,
+      "loss": 0.0261,
+      "step": 1169
+    },
+    {
+      "epoch": 0.03083235562080922,
+      "grad_norm": 0.07775767147541046,
+      "learning_rate": 1.2643221774592518e-05,
+      "loss": 0.0163,
+      "step": 1176
+    },
+    {
+      "epoch": 0.03101588154712356,
+      "grad_norm": 1.1004410982131958,
+      "learning_rate": 1.2125819934742188e-05,
+      "loss": 0.0503,
+      "step": 1183
+    },
+    {
+      "epoch": 0.0311994074734379,
+      "grad_norm": 8.152705192565918,
+      "learning_rate": 1.1617762982099446e-05,
+      "loss": 0.0139,
+      "step": 1190
+    },
+    {
+      "epoch": 0.03138293339975224,
+      "grad_norm": 0.03604988008737564,
+      "learning_rate": 1.1119176272115128e-05,
+      "loss": 0.021,
+      "step": 1197
+    },
+    {
+      "epoch": 0.03156645932606658,
+      "grad_norm": 1.1160600185394287,
+      "learning_rate": 1.0630182823599399e-05,
+      "loss": 0.0177,
+      "step": 1204
+    },
+    {
+      "epoch": 0.03174998525238092,
+      "grad_norm": 4.8045477867126465,
+      "learning_rate": 1.0150903288368741e-05,
+      "loss": 0.0054,
+      "step": 1211
+    },
+    {
+      "epoch": 0.031933511178695265,
+      "grad_norm": 0.6378170251846313,
+      "learning_rate": 9.681455921476839e-06,
+      "loss": 0.0065,
+      "step": 1218
+    },
+    {
+      "epoch": 0.0321170371050096,
+      "grad_norm": 0.3797208070755005,
+      "learning_rate": 9.221956552036992e-06,
+      "loss": 0.003,
+      "step": 1225
+    },
+    {
+      "epoch": 0.03230056303132394,
+      "grad_norm": 0.09285937994718552,
+      "learning_rate": 8.772518554642973e-06,
+      "loss": 0.0036,
+      "step": 1232
+    },
+    {
+      "epoch": 0.03248408895763828,
+      "grad_norm": 0.11193349957466125,
+      "learning_rate": 8.333252821395526e-06,
+      "loss": 0.004,
+      "step": 1239
+    },
+    {
+      "epoch": 0.032667614883952624,
+      "grad_norm": 0.032632287591695786,
+      "learning_rate": 7.904267734541498e-06,
+      "loss": 0.0071,
+      "step": 1246
+    },
+    {
+      "epoch": 0.032851140810266966,
+      "grad_norm": 0.21851947903633118,
+      "learning_rate": 7.485669139732004e-06,
+      "loss": 0.0111,
+      "step": 1253
+    },
+    {
+      "epoch": 0.03303466673658131,
+      "grad_norm": 0.04940886050462723,
+      "learning_rate": 7.077560319906695e-06,
+      "loss": 0.0217,
+      "step": 1260
+    },
+    {
+      "epoch": 0.03321819266289565,
+      "grad_norm": 0.22147436439990997,
+      "learning_rate": 6.680041969810203e-06,
+      "loss": 0.0132,
+      "step": 1267
+    },
+    {
+      "epoch": 0.03340171858920998,
+      "grad_norm": 0.00459816912189126,
+      "learning_rate": 6.293212171147206e-06,
+      "loss": 0.0006,
+      "step": 1274
+    },
+    {
+      "epoch": 0.033585244515524325,
+      "grad_norm": 0.6852213740348816,
+      "learning_rate": 5.917166368382277e-06,
+      "loss": 0.0122,
+      "step": 1281
+    },
+    {
+      "epoch": 0.033768770441838666,
+      "grad_norm": 4.187755584716797,
+      "learning_rate": 5.5519973451903405e-06,
+      "loss": 0.0173,
+      "step": 1288
+    },
+    {
+      "epoch": 0.03395229636815301,
+      "grad_norm": 0.43446940183639526,
+      "learning_rate": 5.197795201563743e-06,
+      "loss": 0.0022,
+      "step": 1295
+    },
+    {
+      "epoch": 0.03413582229446735,
+      "grad_norm": 0.769895613193512,
+      "learning_rate": 4.8546473315813856e-06,
+      "loss": 0.0153,
+      "step": 1302
+    },
+    {
+      "epoch": 0.03431934822078169,
+      "grad_norm": 7.759857654571533,
+      "learning_rate": 4.522638401845547e-06,
+      "loss": 0.0355,
+      "step": 1309
+    },
+    {
+      "epoch": 0.03450287414709603,
+      "grad_norm": 2.7394161224365234,
+      "learning_rate": 4.2018503305916775e-06,
+      "loss": 0.0087,
+      "step": 1316
+    },
+    {
+      "epoch": 0.03468640007341037,
+      "grad_norm": 0.007506331894546747,
+      "learning_rate": 3.892362267476313e-06,
+      "loss": 0.0014,
+      "step": 1323
+    },
+    {
+      "epoch": 0.03486992599972471,
+      "grad_norm": 0.7357063293457031,
+      "learning_rate": 3.5942505740480582e-06,
+      "loss": 0.0022,
+      "step": 1330
+    },
+    {
+      "epoch": 0.03505345192603905,
+      "grad_norm": 0.06958146393299103,
+      "learning_rate": 3.3075888049065196e-06,
+      "loss": 0.0328,
+      "step": 1337
+    },
+    {
+      "epoch": 0.03523697785235339,
+      "grad_norm": 0.009643347933888435,
+      "learning_rate": 3.03244768955383e-06,
+      "loss": 0.0039,
+      "step": 1344
+    },
+    {
+      "epoch": 0.035420503778667733,
+      "grad_norm": 3.819657802581787,
+      "learning_rate": 2.7688951149431595e-06,
+      "loss": 0.0125,
+      "step": 1351
+    },
+    {
+      "epoch": 0.035604029704982075,
+      "grad_norm": 0.1322094202041626,
+      "learning_rate": 2.5169961087286974e-06,
+      "loss": 0.0029,
+      "step": 1358
+    },
+    {
+      "epoch": 0.03578755563129642,
+      "grad_norm": 3.7723782062530518,
+      "learning_rate": 2.276812823220964e-06,
+      "loss": 0.0058,
+      "step": 1365
+    },
+    {
+      "epoch": 0.03597108155761075,
+      "grad_norm": 0.009944421239197254,
+      "learning_rate": 2.048404520051722e-06,
+      "loss": 0.0132,
+      "step": 1372
+    },
+    {
+      "epoch": 0.03615460748392509,
+      "grad_norm": 0.7622977495193481,
+      "learning_rate": 1.8318275555520237e-06,
+      "loss": 0.0005,
+      "step": 1379
+    },
+    {
+      "epoch": 0.036338133410239434,
+      "grad_norm": 0.012031285092234612,
+      "learning_rate": 1.6271353668471655e-06,
+      "loss": 0.0029,
+      "step": 1386
+    },
+    {
+      "epoch": 0.036521659336553776,
+      "grad_norm": 0.008338500745594501,
+      "learning_rate": 1.4343784586718311e-06,
+      "loss": 0.0211,
+      "step": 1393
+    },
+    {
+      "epoch": 0.03670518526286812,
+      "grad_norm": 0.508963942527771,
+      "learning_rate": 1.2536043909088191e-06,
+      "loss": 0.0318,
+      "step": 1400
+    },
+    {
+      "epoch": 0.03688871118918246,
+      "grad_norm": 0.12881743907928467,
+      "learning_rate": 1.0848577668543802e-06,
+      "loss": 0.0045,
+      "step": 1407
+    },
+    {
+      "epoch": 0.0370722371154968,
+      "grad_norm": 0.018107250332832336,
+      "learning_rate": 9.281802222129765e-07,
+      "loss": 0.0005,
+      "step": 1414
+    },
+    {
+      "epoch": 0.03725576304181114,
+      "grad_norm": 0.06451334804296494,
+      "learning_rate": 7.836104148243484e-07,
+      "loss": 0.0039,
+      "step": 1421
+    },
+    {
+      "epoch": 0.03743928896812548,
+      "grad_norm": 1.573056697845459,
+      "learning_rate": 6.511840151252169e-07,
+      "loss": 0.0027,
+      "step": 1428
+    },
+    {
+      "epoch": 0.03762281489443982,
+      "grad_norm": 1.8758277893066406,
+      "learning_rate": 5.309336973481683e-07,
+      "loss": 0.0031,
+      "step": 1435
+    },
+    {
+      "epoch": 0.03780634082075416,
+      "grad_norm": 0.5659865140914917,
+      "learning_rate": 4.228891314597694e-07,
+      "loss": 0.0009,
+      "step": 1442
+    },
+    {
+      "epoch": 0.0379898667470685,
+      "grad_norm": 14.735620498657227,
+      "learning_rate": 3.2707697583995167e-07,
+      "loss": 0.0234,
+      "step": 1449
+    },
+    {
+      "epoch": 0.03817339267338284,
+      "grad_norm": 0.0797620490193367,
+      "learning_rate": 2.4352087070443895e-07,
+      "loss": 0.0138,
+      "step": 1456
+    },
+    {
+      "epoch": 0.038356918599697185,
+      "grad_norm": 0.11197490990161896,
+      "learning_rate": 1.7224143227190236e-07,
+      "loss": 0.0089,
+      "step": 1463
+    },
+    {
+      "epoch": 0.038540444526011526,
+      "grad_norm": 0.23225674033164978,
+      "learning_rate": 1.132562476771959e-07,
+      "loss": 0.0067,
+      "step": 1470
+    },
+    {
+      "epoch": 0.03872397045232586,
+      "grad_norm": 0.05358387902379036,
+      "learning_rate": 6.657987063200533e-08,
+      "loss": 0.1307,
+      "step": 1477
+    },
+    {
+      "epoch": 0.0389074963786402,
+      "grad_norm": 0.02191798947751522,
+      "learning_rate": 3.2223817833931805e-08,
+      "loss": 0.0148,
+      "step": 1484
+    },
+    {
+      "epoch": 0.039091022304954544,
+      "grad_norm": 0.009089218452572823,
+      "learning_rate": 1.019656612492592e-08,
+      "loss": 0.0039,
+      "step": 1491
+    },
+    {
+      "epoch": 0.039274548231268885,
+      "grad_norm": 0.06261090934276581,
+      "learning_rate": 5.035503997385949e-10,
+      "loss": 0.0016,
+      "step": 1498
+    }
+  ],
+  "logging_steps": 7,
+  "max_steps": 1500,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 5.546484091060224e+17,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

last-checkpoint/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a40d0541ee8230cab00c53d4c8d55d6d30c78edb3e956e7910c8ad12a6fff3d4
+size 6776