Training in progress, step 40, checkpoint

last-checkpoint/README.md

@@ -0,0 +1,202 @@
+---
+base_model: JackFram/llama-68m
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.14.0

last-checkpoint/adapter_config.json

@@ -0,0 +1,37 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "JackFram/llama-68m",
+  "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "o_proj",
+    "v_proj",
+    "q_proj",
+    "up_proj",
+    "down_proj",
+    "k_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

last-checkpoint/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3879c6f02d411a9060a269d073d0022e53fb5cd9351db19590d4ee8aeed425aa
+size 18091656

last-checkpoint/optimizer.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8ac70b08e8cc492bc90f894d12a59aabefeae26eb6f1e2f8d8d611266df82c50
-size 4291766
+oid sha256:33e11a8f2aaaf2e13039cdf39771b7f1428ba91c0d74ae33e38129cc700c8a19
+size 36192934

last-checkpoint/rng_state_0.pth

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:20d70fad263b22723fec206cf46c7be2eacd2f78f087c3d0bdf030ac5240ae13
+oid sha256:48f7d3d6417771bc41f14dc3a4e29cf435bb794a2fbe326ff690415536e9f694
 size 15984

last-checkpoint/rng_state_1.pth

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ec3fcb4be00d780df479f9dac4126d0348e75ff289a5063a7ed0574b57fbc9eb
+oid sha256:a78774ea0f9175386adbfcf6da8a43dc4a36e240ce097d48bf11c61c757bdf0e
 size 15984

last-checkpoint/rng_state_2.pth

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ec5b78147ecf7eb0aa33d5383c0a50cc39cc808ebc0f0a52f2c385de2c5c0c79
+oid sha256:ad697120a00356b7a0cca6d572cd222fa73999494592a5e01311fa425f8f683e
 size 15984

last-checkpoint/rng_state_3.pth

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:def24f9a4a2da9e3ecf86804e9ee3edba51a08304bdf5081e4331fade20eec8a
+oid sha256:409650bf679757133c2cd3fc2d2a611f62f48573438f52578dd0dd9946ccdbd8
 size 15984

last-checkpoint/rng_state_4.pth

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d542d427bf2bb6ef4b7b489b3ff8728a8d7a7269befb6dac998921cfec4fa2fe
+oid sha256:b508825a2f3640fb000f0904f66d278bb616305b994e9dbc25894641706a8231
 size 15984

last-checkpoint/rng_state_5.pth

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:23a2b76363f40a9152be338f15b92fe9aea0c0d9d18bae8b15de98ba4365477a
+oid sha256:b0c37773e28fdb852aead7bde72d13489aa95fa4ff50dda40462774f5b2279d0
 size 15984

last-checkpoint/rng_state_6.pth

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e1ca7fe9106d3f63890388b3012f13801b1f0ccf1ae72d0176ee3cc810f72b56
+oid sha256:cbc95533dca0e8a2c91a255cd495452c0e8822e0350bf6e3881450ccbe6dd1df
 size 15984

last-checkpoint/rng_state_7.pth

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:71951a6bb2a69638cdc2957f124934a2a5d71b1ae3bd32982c92819f8437c8ba
+oid sha256:f60f3e9783e041abf20fb5f6a884257de3ed15d3620e3e4f19bd31657bbdc23c
 size 15984

last-checkpoint/scheduler.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7b204b20cdc92a140e2e21e015bdaa04af008c00e0bde30e59edf0f23817a338
+oid sha256:fde4abb1ba5c270405a21e3f71ef7e0ec6bd9f941b0624669b81510eb3186ff6
 size 1064

last-checkpoint/special_tokens_map.json

@@ -2,19 +2,19 @@
   "bos_token": {
     "content": "<s>",
     "lstrip": false,
-    "normalized": false,
+    "normalized": true,
     "rstrip": false,
     "single_word": false
   },
   "eos_token": {
-    "content": "<|endoftext|>",
+    "content": "</s>",
     "lstrip": false,
-    "normalized": false,
+    "normalized": true,
     "rstrip": false,
     "single_word": false
   },
   "pad_token": {
-    "content": "<|endoftext|>",
+    "content": "</s>",
     "lstrip": false,
     "normalized": false,
     "rstrip": false,
@@ -23,7 +23,7 @@
   "unk_token": {
     "content": "<unk>",
     "lstrip": false,
-    "normalized": false,
+    "normalized": true,
     "rstrip": false,
     "single_word": false
   }

last-checkpoint/tokenizer.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b013fe7282b7984bc14b1c64c2a70dd06b652a969810fbba6217f4ac70339f44
-size 3621089
+oid sha256:f1a70193a38a63cf5d5f51f6b2b989eb77af06336171584533c4c6012fc06894
+size 3619014

last-checkpoint/tokenizer_config.json

@@ -6,7 +6,7 @@
     "0": {
       "content": "<unk>",
       "lstrip": false,
-      "normalized": false,
+      "normalized": true,
       "rstrip": false,
       "single_word": false,
       "special": true
@@ -14,7 +14,7 @@
     "1": {
       "content": "<s>",
       "lstrip": false,
-      "normalized": false,
+      "normalized": true,
       "rstrip": false,
       "single_word": false,
       "special": true
@@ -23,109 +23,21 @@
       "content": "</s>",
       "lstrip": false,
       "normalized": false,
-      "rstrip": true,
-      "single_word": false,
-      "special": false
-    },
-    "32000": {
-      "content": "<|endoftext|>",
-      "lstrip": false,
-      "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
-    },
-    "32001": {
-      "content": "<|assistant|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": true,
-      "single_word": false,
-      "special": true
-    },
-    "32002": {
-      "content": "<|placeholder1|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": true,
-      "single_word": false,
-      "special": true
-    },
-    "32003": {
-      "content": "<|placeholder2|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": true,
-      "single_word": false,
-      "special": true
-    },
-    "32004": {
-      "content": "<|placeholder3|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": true,
-      "single_word": false,
-      "special": true
-    },
-    "32005": {
-      "content": "<|placeholder4|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": true,
-      "single_word": false,
-      "special": true
-    },
-    "32006": {
-      "content": "<|system|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": true,
-      "single_word": false,
-      "special": true
-    },
-    "32007": {
-      "content": "<|end|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": true,
-      "single_word": false,
-      "special": true
-    },
-    "32008": {
-      "content": "<|placeholder5|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": true,
-      "single_word": false,
-      "special": true
-    },
-    "32009": {
-      "content": "<|placeholder6|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": true,
-      "single_word": false,
-      "special": true
-    },
-    "32010": {
-      "content": "<|user|>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": true,
-      "single_word": false,
-      "special": true
     }
   },
   "bos_token": "<s>",
-  "chat_template": "{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') %}{{'<|user|>' + '\n' + message['content'] + '<|end|>' + '\n' + '<|assistant|>' + '\n'}}{% elif (message['role'] == 'assistant') %}{{message['content'] + '<|end|>' + '\n'}}{% endif %}{% endfor %}",
+  "chat_template": "{% for message in messages %}{% if message['role'] == 'user' %}{{ '### Instruction: ' + message['content'] + '\n\n' }}{% elif message['role'] == 'assistant' %}{{ '### Response: ' + message['content'] + eos_token}}{% endif %}{% endfor %}",
   "clean_up_tokenization_spaces": false,
-  "eos_token": "<|endoftext|>",
+  "eos_token": "</s>",
   "extra_special_tokens": {},
   "legacy": true,
-  "model_max_length": 4096,
-  "pad_token": "<|endoftext|>",
-  "padding_side": "left",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
   "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
   "tokenizer_class": "LlamaTokenizer",
   "unk_token": "<unk>",
   "use_default_system_prompt": false,

last-checkpoint/trainer_state.json

@@ -1,1873 +1,55 @@
 {
   "best_metric": null,
   "best_model_checkpoint": null,
-  "epoch": 0.7398638650488311,
+  "epoch": 0.7582938388625592,
   "eval_steps": 200,
-  "global_step": 2500,
+  "global_step": 40,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
   "log_history": [
     {
-      "epoch": 0.0002959455460195324,
-      "eval_loss": 10.377325057983398,
-      "eval_runtime": 11.0363,
-      "eval_samples_per_second": 136.096,
-      "eval_steps_per_second": 34.069,
+      "epoch": 0.018957345971563982,
+      "eval_loss": 4.226168632507324,
+      "eval_runtime": 1.4697,
+      "eval_samples_per_second": 1021.29,
+      "eval_steps_per_second": 127.916,
       "step": 1
     },
     {
-      "epoch": 0.002959455460195324,
-      "grad_norm": 0.43359375,
-      "learning_rate": 1.6000000000000003e-05,
-      "loss": 10.3751,
+      "epoch": 0.1895734597156398,
+      "grad_norm": 1.3090990781784058,
+      "learning_rate": 2.6666666666666667e-05,
+      "loss": 3.5627,
       "step": 10
     },
     {
-      "epoch": 0.005918910920390648,
-      "grad_norm": 0.51171875,
-      "learning_rate": 3.2000000000000005e-05,
-      "loss": 10.3738,
+      "epoch": 0.3791469194312796,
+      "grad_norm": 0.8479055762290955,
+      "learning_rate": 5.333333333333333e-05,
+      "loss": 3.5438,
       "step": 20
     },
     {
-      "epoch": 0.008878366380585973,
-      "grad_norm": 0.640625,
-      "learning_rate": 4.8e-05,
-      "loss": 10.3766,
+      "epoch": 0.5687203791469194,
+      "grad_norm": 1.6812483072280884,
+      "learning_rate": 8e-05,
+      "loss": 3.3698,
       "step": 30
     },
     {
-      "epoch": 0.011837821840781295,
-      "grad_norm": 0.8359375,
-      "learning_rate": 6.400000000000001e-05,
-      "loss": 10.3761,
+      "epoch": 0.7582938388625592,
+      "grad_norm": 0.815318763256073,
+      "learning_rate": 0.00010666666666666667,
+      "loss": 3.2253,
       "step": 40
-    },
-    {
-      "epoch": 0.01479727730097662,
-      "grad_norm": 1.5859375,
-      "learning_rate": 8e-05,
-      "loss": 10.369,
-      "step": 50
-    },
-    {
-      "epoch": 0.017756732761171946,
-      "grad_norm": 0.462890625,
-      "learning_rate": 9.6e-05,
-      "loss": 10.3704,
-      "step": 60
-    },
-    {
-      "epoch": 0.020716188221367268,
-      "grad_norm": 0.6015625,
-      "learning_rate": 0.00011200000000000001,
-      "loss": 10.362,
-      "step": 70
-    },
-    {
-      "epoch": 0.02367564368156259,
-      "grad_norm": 0.76171875,
-      "learning_rate": 0.00012800000000000002,
-      "loss": 10.3539,
-      "step": 80
-    },
-    {
-      "epoch": 0.026635099141757917,
-      "grad_norm": 1.1484375,
-      "learning_rate": 0.000144,
-      "loss": 10.3111,
-      "step": 90
-    },
-    {
-      "epoch": 0.02959455460195324,
-      "grad_norm": 2.53125,
-      "learning_rate": 0.00016,
-      "loss": 10.2556,
-      "step": 100
-    },
-    {
-      "epoch": 0.032554010062148565,
-      "grad_norm": 0.7578125,
-      "learning_rate": 0.00017600000000000002,
-      "loss": 10.1628,
-      "step": 110
-    },
-    {
-      "epoch": 0.03551346552234389,
-      "grad_norm": 0.66796875,
-      "learning_rate": 0.000192,
-      "loss": 10.0397,
-      "step": 120
-    },
-    {
-      "epoch": 0.03847292098253921,
-      "grad_norm": 0.7421875,
-      "learning_rate": 0.0001999978128380225,
-      "loss": 9.9248,
-      "step": 130
-    },
-    {
-      "epoch": 0.041432376442734536,
-      "grad_norm": 0.88671875,
-      "learning_rate": 0.0001999803161162393,
-      "loss": 9.8201,
-      "step": 140
-    },
-    {
-      "epoch": 0.04439183190292986,
-      "grad_norm": 1.78125,
-      "learning_rate": 0.00019994532573409262,
-      "loss": 9.7523,
-      "step": 150
-    },
-    {
-      "epoch": 0.04735128736312518,
-      "grad_norm": 0.609375,
-      "learning_rate": 0.00019989284781388617,
-      "loss": 9.6134,
-      "step": 160
-    },
-    {
-      "epoch": 0.05031074282332051,
-      "grad_norm": 0.57421875,
-      "learning_rate": 0.00019982289153773646,
-      "loss": 9.5326,
-      "step": 170
-    },
-    {
-      "epoch": 0.053270198283515834,
-      "grad_norm": 0.6640625,
-      "learning_rate": 0.00019973546914596623,
-      "loss": 9.4273,
-      "step": 180
-    },
-    {
-      "epoch": 0.05622965374371116,
-      "grad_norm": 0.90234375,
-      "learning_rate": 0.00019963059593496268,
-      "loss": 9.3176,
-      "step": 190
-    },
-    {
-      "epoch": 0.05918910920390648,
-      "grad_norm": 1.6015625,
-      "learning_rate": 0.00019950829025450114,
-      "loss": 9.2412,
-      "step": 200
-    },
-    {
-      "epoch": 0.05918910920390648,
-      "eval_loss": 9.224679946899414,
-      "eval_runtime": 11.174,
-      "eval_samples_per_second": 134.419,
-      "eval_steps_per_second": 33.649,
-      "step": 200
-    },
-    {
-      "epoch": 0.062148564664101805,
-      "grad_norm": 0.640625,
-      "learning_rate": 0.0001993685735045343,
-      "loss": 9.1904,
-      "step": 210
-    },
-    {
-      "epoch": 0.06510802012429713,
-      "grad_norm": 0.71484375,
-      "learning_rate": 0.0001992114701314478,
-      "loss": 9.1332,
-      "step": 220
-    },
-    {
-      "epoch": 0.06806747558449246,
-      "grad_norm": 0.703125,
-      "learning_rate": 0.000199037007623783,
-      "loss": 9.0845,
-      "step": 230
-    },
-    {
-      "epoch": 0.07102693104468778,
-      "grad_norm": 0.83984375,
-      "learning_rate": 0.00019884521650742715,
-      "loss": 8.992,
-      "step": 240
-    },
-    {
-      "epoch": 0.0739863865048831,
-      "grad_norm": 2.59375,
-      "learning_rate": 0.00019863613034027224,
-      "loss": 9.0056,
-      "step": 250
-    },
-    {
-      "epoch": 0.07694584196507842,
-      "grad_norm": 0.6953125,
-      "learning_rate": 0.0001984097857063434,
-      "loss": 8.8204,
-      "step": 260
-    },
-    {
-      "epoch": 0.07990529742527375,
-      "grad_norm": 0.77734375,
-      "learning_rate": 0.0001981662222093976,
-      "loss": 8.8472,
-      "step": 270
-    },
-    {
-      "epoch": 0.08286475288546907,
-      "grad_norm": 0.70703125,
-      "learning_rate": 0.00019790548246599447,
-      "loss": 8.785,
-      "step": 280
-    },
-    {
-      "epoch": 0.0858242083456644,
-      "grad_norm": 0.8515625,
-      "learning_rate": 0.00019762761209803927,
-      "loss": 8.7212,
-      "step": 290
-    },
-    {
-      "epoch": 0.08878366380585972,
-      "grad_norm": 1.6484375,
-      "learning_rate": 0.0001973326597248006,
-      "loss": 8.748,
-      "step": 300
-    },
-    {
-      "epoch": 0.09174311926605505,
-      "grad_norm": 0.609375,
-      "learning_rate": 0.00019702067695440332,
-      "loss": 8.6333,
-      "step": 310
-    },
-    {
-      "epoch": 0.09470257472625036,
-      "grad_norm": 0.6171875,
-      "learning_rate": 0.00019669171837479873,
-      "loss": 8.6464,
-      "step": 320
-    },
-    {
-      "epoch": 0.09766203018644569,
-      "grad_norm": 0.640625,
-      "learning_rate": 0.00019634584154421317,
-      "loss": 8.6169,
-      "step": 330
-    },
-    {
-      "epoch": 0.10062148564664102,
-      "grad_norm": 0.79296875,
-      "learning_rate": 0.00019598310698107702,
-      "loss": 8.541,
-      "step": 340
-    },
-    {
-      "epoch": 0.10358094110683634,
-      "grad_norm": 1.8203125,
-      "learning_rate": 0.00019560357815343577,
-      "loss": 8.4839,
-      "step": 350
-    },
-    {
-      "epoch": 0.10654039656703167,
-      "grad_norm": 0.77734375,
-      "learning_rate": 0.00019520732146784491,
-      "loss": 8.5185,
-      "step": 360
-    },
-    {
-      "epoch": 0.109499852027227,
-      "grad_norm": 1.0703125,
-      "learning_rate": 0.0001947944062577507,
-      "loss": 8.4832,
-      "step": 370
-    },
-    {
-      "epoch": 0.11245930748742232,
-      "grad_norm": 0.734375,
-      "learning_rate": 0.00019436490477135878,
-      "loss": 8.4174,
-      "step": 380
-    },
-    {
-      "epoch": 0.11541876294761765,
-      "grad_norm": 0.84375,
-      "learning_rate": 0.00019391889215899299,
-      "loss": 8.477,
-      "step": 390
-    },
-    {
-      "epoch": 0.11837821840781296,
-      "grad_norm": 1.9921875,
-      "learning_rate": 0.0001934564464599461,
-      "loss": 8.2947,
-      "step": 400
-    },
-    {
-      "epoch": 0.11837821840781296,
-      "eval_loss": 8.401090621948242,
-      "eval_runtime": 12.2623,
-      "eval_samples_per_second": 122.489,
-      "eval_steps_per_second": 30.663,
-      "step": 400
-    },
-    {
-      "epoch": 0.12133767386800828,
-      "grad_norm": 0.58984375,
-      "learning_rate": 0.00019297764858882514,
-      "loss": 8.421,
-      "step": 410
-    },
-    {
-      "epoch": 0.12429712932820361,
-      "grad_norm": 0.62109375,
-      "learning_rate": 0.00019248258232139388,
-      "loss": 8.3741,
-      "step": 420
-    },
-    {
-      "epoch": 0.12725658478839894,
-      "grad_norm": 0.8203125,
-      "learning_rate": 0.00019197133427991436,
-      "loss": 8.4248,
-      "step": 430
-    },
-    {
-      "epoch": 0.13021604024859426,
-      "grad_norm": 1.015625,
-      "learning_rate": 0.00019144399391799043,
-      "loss": 8.3223,
-      "step": 440
-    },
-    {
-      "epoch": 0.1331754957087896,
-      "grad_norm": 1.6796875,
-      "learning_rate": 0.00019090065350491626,
-      "loss": 8.2908,
-      "step": 450
-    },
-    {
-      "epoch": 0.1361349511689849,
-      "grad_norm": 0.671875,
-      "learning_rate": 0.0001903414081095315,
-      "loss": 8.3506,
-      "step": 460
-    },
-    {
-      "epoch": 0.13909440662918024,
-      "grad_norm": 0.65234375,
-      "learning_rate": 0.00018976635558358722,
-      "loss": 8.2671,
-      "step": 470
-    },
-    {
-      "epoch": 0.14205386208937557,
-      "grad_norm": 0.7578125,
-      "learning_rate": 0.00018917559654462474,
-      "loss": 8.2734,
-      "step": 480
-    },
-    {
-      "epoch": 0.1450133175495709,
-      "grad_norm": 0.765625,
-      "learning_rate": 0.00018856923435837022,
-      "loss": 8.1702,
-      "step": 490
-    },
-    {
-      "epoch": 0.1479727730097662,
-      "grad_norm": 1.359375,
-      "learning_rate": 0.0001879473751206489,
-      "loss": 8.2463,
-      "step": 500
-    },
-    {
-      "epoch": 0.15093222846996152,
-      "grad_norm": 0.609375,
-      "learning_rate": 0.00018731012763882133,
-      "loss": 8.1803,
-      "step": 510
-    },
-    {
-      "epoch": 0.15389168393015684,
-      "grad_norm": 0.65234375,
-      "learning_rate": 0.00018665760341274505,
-      "loss": 8.1842,
-      "step": 520
-    },
-    {
-      "epoch": 0.15685113939035217,
-      "grad_norm": 0.66015625,
-      "learning_rate": 0.00018598991661526572,
-      "loss": 8.2101,
-      "step": 530
-    },
-    {
-      "epoch": 0.1598105948505475,
-      "grad_norm": 0.78125,
-      "learning_rate": 0.00018530718407223974,
-      "loss": 8.3214,
-      "step": 540
-    },
-    {
-      "epoch": 0.16277005031074282,
-      "grad_norm": 1.75,
-      "learning_rate": 0.00018460952524209355,
-      "loss": 8.1588,
-      "step": 550
-    },
-    {
-      "epoch": 0.16572950577093815,
-      "grad_norm": 0.6015625,
-      "learning_rate": 0.00018389706219492147,
-      "loss": 8.2532,
-      "step": 560
-    },
-    {
-      "epoch": 0.16868896123113347,
-      "grad_norm": 0.625,
-      "learning_rate": 0.00018316991959112716,
-      "loss": 8.1023,
-      "step": 570
-    },
-    {
-      "epoch": 0.1716484166913288,
-      "grad_norm": 0.69140625,
-      "learning_rate": 0.00018242822465961176,
-      "loss": 8.1897,
-      "step": 580
-    },
-    {
-      "epoch": 0.17460787215152412,
-      "grad_norm": 0.78125,
-      "learning_rate": 0.00018167210717551224,
-      "loss": 8.1359,
-      "step": 590
-    },
-    {
-      "epoch": 0.17756732761171945,
-      "grad_norm": 1.9140625,
-      "learning_rate": 0.00018090169943749476,
-      "loss": 8.107,
-      "step": 600
-    },
-    {
-      "epoch": 0.17756732761171945,
-      "eval_loss": 8.149062156677246,
-      "eval_runtime": 10.4771,
-      "eval_samples_per_second": 143.36,
-      "eval_steps_per_second": 35.888,
-      "step": 600
-    },
-    {
-      "epoch": 0.18052678307191478,
-      "grad_norm": 0.73828125,
-      "learning_rate": 0.00018011713624460608,
-      "loss": 8.1806,
-      "step": 610
-    },
-    {
-      "epoch": 0.1834862385321101,
-      "grad_norm": 0.69140625,
-      "learning_rate": 0.00017931855487268782,
-      "loss": 8.1251,
-      "step": 620
-    },
-    {
-      "epoch": 0.18644569399230543,
-      "grad_norm": 0.75,
-      "learning_rate": 0.0001785060950503568,
-      "loss": 8.2358,
-      "step": 630
-    },
-    {
-      "epoch": 0.18940514945250073,
-      "grad_norm": 0.9453125,
-      "learning_rate": 0.00017767989893455698,
-      "loss": 7.983,
-      "step": 640
-    },
-    {
-      "epoch": 0.19236460491269605,
-      "grad_norm": 1.25,
-      "learning_rate": 0.00017684011108568592,
-      "loss": 8.0092,
-      "step": 650
-    },
-    {
-      "epoch": 0.19532406037289138,
-      "grad_norm": 0.63671875,
-      "learning_rate": 0.00017598687844230088,
-      "loss": 8.086,
-      "step": 660
-    },
-    {
-      "epoch": 0.1982835158330867,
-      "grad_norm": 0.5703125,
-      "learning_rate": 0.00017512035029540885,
-      "loss": 8.0729,
-      "step": 670
-    },
-    {
-      "epoch": 0.20124297129328203,
-      "grad_norm": 0.6796875,
-      "learning_rate": 0.000174240678262345,
-      "loss": 8.1146,
-      "step": 680
-    },
-    {
-      "epoch": 0.20420242675347736,
-      "grad_norm": 0.81640625,
-      "learning_rate": 0.000173348016260244,
-      "loss": 8.1021,
-      "step": 690
-    },
-    {
-      "epoch": 0.20716188221367268,
-      "grad_norm": 1.8984375,
-      "learning_rate": 0.00017244252047910892,
-      "loss": 8.0097,
-      "step": 700
-    },
-    {
-      "epoch": 0.210121337673868,
-      "grad_norm": 0.5859375,
-      "learning_rate": 0.00017152434935448256,
-      "loss": 8.0561,
-      "step": 710
-    },
-    {
-      "epoch": 0.21308079313406333,
-      "grad_norm": 0.62890625,
-      "learning_rate": 0.0001705936635397259,
-      "loss": 8.1001,
-      "step": 720
-    },
-    {
-      "epoch": 0.21604024859425866,
-      "grad_norm": 0.73828125,
-      "learning_rate": 0.00016965062587790823,
-      "loss": 8.1157,
-      "step": 730
-    },
-    {
-      "epoch": 0.218999704054454,
-      "grad_norm": 0.73046875,
-      "learning_rate": 0.00016869540137331445,
-      "loss": 8.0555,
-      "step": 740
-    },
-    {
-      "epoch": 0.2219591595146493,
-      "grad_norm": 1.4921875,
-      "learning_rate": 0.00016772815716257412,
-      "loss": 8.0423,
-      "step": 750
-    },
-    {
-      "epoch": 0.22491861497484464,
-      "grad_norm": 0.6484375,
-      "learning_rate": 0.00016674906248541726,
-      "loss": 8.058,
-      "step": 760
-    },
-    {
-      "epoch": 0.22787807043503996,
-      "grad_norm": 0.90625,
-      "learning_rate": 0.00016575828865506245,
-      "loss": 8.0314,
-      "step": 770
-    },
-    {
-      "epoch": 0.2308375258952353,
-      "grad_norm": 0.5703125,
-      "learning_rate": 0.0001647560090282419,
-      "loss": 8.1355,
-      "step": 780
-    },
-    {
-      "epoch": 0.2337969813554306,
-      "grad_norm": 0.94140625,
-      "learning_rate": 0.000163742398974869,
-      "loss": 8.0106,
-      "step": 790
-    },
-    {
-      "epoch": 0.23675643681562591,
-      "grad_norm": 1.984375,
-      "learning_rate": 0.0001627176358473537,
-      "loss": 8.0544,
-      "step": 800
-    },
-    {
-      "epoch": 0.23675643681562591,
-      "eval_loss": 8.075135231018066,
-      "eval_runtime": 11.05,
-      "eval_samples_per_second": 135.927,
-      "eval_steps_per_second": 34.027,
-      "step": 800
-    },
-    {
-      "epoch": 0.23971589227582124,
-      "grad_norm": 0.6015625,
-      "learning_rate": 0.0001616818989495711,
-      "loss": 8.1535,
-      "step": 810
-    },
-    {
-      "epoch": 0.24267534773601657,
-      "grad_norm": 0.57421875,
-      "learning_rate": 0.00016063536950548826,
-      "loss": 8.1044,
-      "step": 820
-    },
-    {
-      "epoch": 0.2456348031962119,
-      "grad_norm": 0.64453125,
-      "learning_rate": 0.0001595782306274553,
-      "loss": 8.0982,
-      "step": 830
-    },
-    {
-      "epoch": 0.24859425865640722,
-      "grad_norm": 0.859375,
-      "learning_rate": 0.00015851066728416618,
-      "loss": 8.0706,
-      "step": 840
-    },
-    {
-      "epoch": 0.25155371411660254,
-      "grad_norm": 1.8359375,
-      "learning_rate": 0.00015743286626829437,
-      "loss": 8.087,
-      "step": 850
-    },
-    {
-      "epoch": 0.25451316957679787,
-      "grad_norm": 0.5625,
-      "learning_rate": 0.00015634501616380967,
-      "loss": 8.0893,
-      "step": 860
-    },
-    {
-      "epoch": 0.2574726250369932,
-      "grad_norm": 0.66796875,
-      "learning_rate": 0.00015524730731298134,
-      "loss": 8.0355,
-      "step": 870
-    },
-    {
-      "epoch": 0.2604320804971885,
-      "grad_norm": 0.65234375,
-      "learning_rate": 0.0001541399317830738,
-      "loss": 8.0417,
-      "step": 880
-    },
-    {
-      "epoch": 0.26339153595738385,
-      "grad_norm": 1.1484375,
-      "learning_rate": 0.0001530230833327405,
-      "loss": 8.1504,
-      "step": 890
-    },
-    {
-      "epoch": 0.2663509914175792,
-      "grad_norm": 1.3046875,
-      "learning_rate": 0.00015189695737812152,
-      "loss": 7.8906,
-      "step": 900
-    },
-    {
-      "epoch": 0.2693104468777745,
-      "grad_norm": 0.57421875,
-      "learning_rate": 0.0001507617509586517,
-      "loss": 8.1158,
-      "step": 910
-    },
-    {
-      "epoch": 0.2722699023379698,
-      "grad_norm": 0.59765625,
-      "learning_rate": 0.00014961766270258422,
-      "loss": 8.0091,
-      "step": 920
-    },
-    {
-      "epoch": 0.27522935779816515,
-      "grad_norm": 0.6484375,
-      "learning_rate": 0.00014846489279223652,
-      "loss": 8.0552,
-      "step": 930
-    },
-    {
-      "epoch": 0.2781888132583605,
-      "grad_norm": 0.81640625,
-      "learning_rate": 0.0001473036429289641,
-      "loss": 8.0668,
-      "step": 940
-    },
-    {
-      "epoch": 0.2811482687185558,
-      "grad_norm": 1.609375,
-      "learning_rate": 0.0001461341162978688,
-      "loss": 7.9344,
-      "step": 950
-    },
-    {
-      "epoch": 0.28410772417875113,
-      "grad_norm": 0.5234375,
-      "learning_rate": 0.00014495651753224705,
-      "loss": 8.0812,
-      "step": 960
-    },
-    {
-      "epoch": 0.28706717963894646,
-      "grad_norm": 0.61328125,
-      "learning_rate": 0.00014377105267778518,
-      "loss": 8.1065,
-      "step": 970
-    },
-    {
-      "epoch": 0.2900266350991418,
-      "grad_norm": 0.71484375,
-      "learning_rate": 0.00014257792915650728,
-      "loss": 8.0159,
-      "step": 980
-    },
-    {
-      "epoch": 0.2929860905593371,
-      "grad_norm": 0.859375,
-      "learning_rate": 0.00014137735573048233,
-      "loss": 8.0729,
-      "step": 990
-    },
-    {
-      "epoch": 0.2959455460195324,
-      "grad_norm": 1.5625,
-      "learning_rate": 0.00014016954246529696,
-      "loss": 7.8343,
-      "step": 1000
-    },
-    {
-      "epoch": 0.2959455460195324,
-      "eval_loss": 8.051301002502441,
-      "eval_runtime": 11.0694,
-      "eval_samples_per_second": 135.69,
-      "eval_steps_per_second": 33.968,
-      "step": 1000
-    },
-    {
-      "epoch": 0.2989050014797277,
-      "grad_norm": 0.734375,
-      "learning_rate": 0.00013895470069330004,
-      "loss": 8.1412,
-      "step": 1010
-    },
-    {
-      "epoch": 0.30186445693992303,
-      "grad_norm": 0.62109375,
-      "learning_rate": 0.00013773304297662559,
-      "loss": 8.0217,
-      "step": 1020
-    },
-    {
-      "epoch": 0.30482391240011836,
-      "grad_norm": 0.890625,
-      "learning_rate": 0.00013650478307000057,
-      "loss": 8.0912,
-      "step": 1030
-    },
-    {
-      "epoch": 0.3077833678603137,
-      "grad_norm": 1.09375,
-      "learning_rate": 0.00013527013588334415,
-      "loss": 8.0892,
-      "step": 1040
-    },
-    {
-      "epoch": 0.310742823320509,
-      "grad_norm": 1.8125,
-      "learning_rate": 0.00013402931744416433,
-      "loss": 8.0011,
-      "step": 1050
-    },
-    {
-      "epoch": 0.31370227878070434,
-      "grad_norm": 0.5703125,
-      "learning_rate": 0.00013278254485975976,
-      "loss": 8.0609,
-      "step": 1060
-    },
-    {
-      "epoch": 0.31666173424089966,
-      "grad_norm": 0.7890625,
-      "learning_rate": 0.00013153003627923218,
-      "loss": 8.1007,
-      "step": 1070
-    },
-    {
-      "epoch": 0.319621189701095,
-      "grad_norm": 0.75390625,
-      "learning_rate": 0.00013027201085531634,
-      "loss": 8.1113,
-      "step": 1080
-    },
-    {
-      "epoch": 0.3225806451612903,
-      "grad_norm": 0.83984375,
-      "learning_rate": 0.00012900868870603503,
-      "loss": 8.1575,
-      "step": 1090
-    },
-    {
-      "epoch": 0.32554010062148564,
-      "grad_norm": 1.5234375,
-      "learning_rate": 0.00012774029087618446,
-      "loss": 8.1575,
-      "step": 1100
-    },
-    {
-      "epoch": 0.32849955608168097,
-      "grad_norm": 0.58984375,
-      "learning_rate": 0.00012646703929865817,
-      "loss": 8.0385,
-      "step": 1110
-    },
-    {
-      "epoch": 0.3314590115418763,
-      "grad_norm": 0.609375,
-      "learning_rate": 0.00012518915675561483,
-      "loss": 7.9723,
-      "step": 1120
-    },
-    {
-      "epoch": 0.3344184670020716,
-      "grad_norm": 0.76171875,
-      "learning_rate": 0.00012390686683949798,
-      "loss": 7.9626,
-      "step": 1130
-    },
-    {
-      "epoch": 0.33737792246226694,
-      "grad_norm": 0.97265625,
-      "learning_rate": 0.00012262039391391404,
-      "loss": 8.0115,
-      "step": 1140
-    },
-    {
-      "epoch": 0.34033737792246227,
-      "grad_norm": 2.328125,
-      "learning_rate": 0.0001213299630743747,
-      "loss": 8.0105,
-      "step": 1150
-    },
-    {
-      "epoch": 0.3432968333826576,
-      "grad_norm": 0.60546875,
-      "learning_rate": 0.00012003580010891213,
-      "loss": 8.0462,
-      "step": 1160
-    },
-    {
-      "epoch": 0.3462562888428529,
-      "grad_norm": 0.7109375,
-      "learning_rate": 0.00011873813145857249,
-      "loss": 8.0028,
-      "step": 1170
-    },
-    {
-      "epoch": 0.34921574430304825,
-      "grad_norm": 0.65625,
-      "learning_rate": 0.00011743718417779517,
-      "loss": 8.1166,
-      "step": 1180
-    },
-    {
-      "epoch": 0.3521751997632436,
-      "grad_norm": 1.3046875,
-      "learning_rate": 0.00011613318589468511,
-      "loss": 7.9591,
-      "step": 1190
-    },
-    {
-      "epoch": 0.3551346552234389,
-      "grad_norm": 1.125,
-      "learning_rate": 0.0001148263647711842,
-      "loss": 7.9614,
-      "step": 1200
-    },
-    {
-      "epoch": 0.3551346552234389,
-      "eval_loss": 8.032776832580566,
-      "eval_runtime": 11.1602,
-      "eval_samples_per_second": 134.585,
-      "eval_steps_per_second": 33.691,
-      "step": 1200
-    },
-    {
-      "epoch": 0.3580941106836342,
-      "grad_norm": 0.58984375,
-      "learning_rate": 0.0001135169494631497,
-      "loss": 8.0268,
-      "step": 1210
-    },
-    {
-      "epoch": 0.36105356614382955,
-      "grad_norm": 0.6328125,
-      "learning_rate": 0.00011220516908034601,
-      "loss": 8.0724,
-      "step": 1220
-    },
-    {
-      "epoch": 0.3640130216040249,
-      "grad_norm": 0.65234375,
-      "learning_rate": 0.00011089125314635726,
-      "loss": 8.0887,
-      "step": 1230
-    },
-    {
-      "epoch": 0.3669724770642202,
-      "grad_norm": 0.83984375,
-      "learning_rate": 0.00010957543155842702,
-      "loss": 8.1362,
-      "step": 1240
-    },
-    {
-      "epoch": 0.36993193252441553,
-      "grad_norm": 1.1953125,
-      "learning_rate": 0.00010825793454723325,
-      "loss": 8.0316,
-      "step": 1250
-    },
-    {
-      "epoch": 0.37289138798461086,
-      "grad_norm": 0.64453125,
-      "learning_rate": 0.00010693899263660441,
-      "loss": 8.0103,
-      "step": 1260
-    },
-    {
-      "epoch": 0.3758508434448062,
-      "grad_norm": 0.62109375,
-      "learning_rate": 0.00010561883660318455,
-      "loss": 8.0047,
-      "step": 1270
-    },
-    {
-      "epoch": 0.37881029890500145,
-      "grad_norm": 0.6953125,
-      "learning_rate": 0.00010429769743605407,
-      "loss": 8.0424,
-      "step": 1280
-    },
-    {
-      "epoch": 0.3817697543651968,
-      "grad_norm": 0.90625,
-      "learning_rate": 0.00010297580629631325,
-      "loss": 7.9585,
-      "step": 1290
-    },
-    {
-      "epoch": 0.3847292098253921,
-      "grad_norm": 1.4296875,
-      "learning_rate": 0.00010165339447663587,
-      "loss": 7.8657,
-      "step": 1300
-    },
-    {
-      "epoch": 0.38768866528558743,
-      "grad_norm": 0.5546875,
-      "learning_rate": 0.00010033069336079952,
-      "loss": 8.115,
-      "step": 1310
-    },
-    {
-      "epoch": 0.39064812074578276,
-      "grad_norm": 0.640625,
-      "learning_rate": 9.900793438320037e-05,
-      "loss": 8.0227,
-      "step": 1320
-    },
-    {
-      "epoch": 0.3936075762059781,
-      "grad_norm": 0.6796875,
-      "learning_rate": 9.768534898835862e-05,
-      "loss": 8.0111,
-      "step": 1330
-    },
-    {
-      "epoch": 0.3965670316661734,
-      "grad_norm": 0.8203125,
-      "learning_rate": 9.636316859042259e-05,
-      "loss": 8.005,
-      "step": 1340
-    },
-    {
-      "epoch": 0.39952648712636873,
-      "grad_norm": 1.1640625,
-      "learning_rate": 9.504162453267777e-05,
-      "loss": 7.9406,
-      "step": 1350
-    },
-    {
-      "epoch": 0.40248594258656406,
-      "grad_norm": 0.5859375,
-      "learning_rate": 9.372094804706867e-05,
-      "loss": 8.0917,
-      "step": 1360
-    },
-    {
-      "epoch": 0.4054453980467594,
-      "grad_norm": 0.69921875,
-      "learning_rate": 9.24013702137397e-05,
-      "loss": 8.0005,
-      "step": 1370
-    },
-    {
-      "epoch": 0.4084048535069547,
-      "grad_norm": 0.78515625,
-      "learning_rate": 9.108312192060298e-05,
-      "loss": 8.0312,
-      "step": 1380
-    },
-    {
-      "epoch": 0.41136430896715004,
-      "grad_norm": 1.234375,
-      "learning_rate": 8.97664338229395e-05,
-      "loss": 8.0493,
-      "step": 1390
-    },
-    {
-      "epoch": 0.41432376442734536,
-      "grad_norm": 1.9375,
-      "learning_rate": 8.845153630304139e-05,
-      "loss": 8.0718,
-      "step": 1400
-    },
-    {
-      "epoch": 0.41432376442734536,
-      "eval_loss": 8.029215812683105,
-      "eval_runtime": 11.3199,
-      "eval_samples_per_second": 132.687,
-      "eval_steps_per_second": 33.216,
-      "step": 1400
-    },
-    {
-      "epoch": 0.4172832198875407,
-      "grad_norm": 0.578125,
-      "learning_rate": 8.713865942990141e-05,
-      "loss": 8.0277,
-      "step": 1410
-    },
-    {
-      "epoch": 0.420242675347736,
-      "grad_norm": 0.72265625,
-      "learning_rate": 8.582803291895758e-05,
-      "loss": 8.0412,
-      "step": 1420
-    },
-    {
-      "epoch": 0.42320213080793134,
-      "grad_norm": 0.6875,
-      "learning_rate": 8.451988609189987e-05,
-      "loss": 8.1284,
-      "step": 1430
-    },
-    {
-      "epoch": 0.42616158626812667,
-      "grad_norm": 0.94921875,
-      "learning_rate": 8.321444783654524e-05,
-      "loss": 8.014,
-      "step": 1440
-    },
-    {
-      "epoch": 0.429121041728322,
-      "grad_norm": 1.6875,
-      "learning_rate": 8.191194656678904e-05,
-      "loss": 8.0557,
-      "step": 1450
-    },
-    {
-      "epoch": 0.4320804971885173,
-      "grad_norm": 0.6875,
-      "learning_rate": 8.061261018263919e-05,
-      "loss": 8.0064,
-      "step": 1460
-    },
-    {
-      "epoch": 0.43503995264871265,
-      "grad_norm": 0.7109375,
-      "learning_rate": 7.931666603034033e-05,
-      "loss": 7.99,
-      "step": 1470
-    },
-    {
-      "epoch": 0.437999408108908,
-      "grad_norm": 1.1328125,
-      "learning_rate": 7.80243408625947e-05,
-      "loss": 8.0039,
-      "step": 1480
-    },
-    {
-      "epoch": 0.4409588635691033,
-      "grad_norm": 0.9921875,
-      "learning_rate": 7.673586079888698e-05,
-      "loss": 8.0759,
-      "step": 1490
-    },
-    {
-      "epoch": 0.4439183190292986,
-      "grad_norm": 1.3984375,
-      "learning_rate": 7.54514512859201e-05,
-      "loss": 7.8971,
-      "step": 1500
-    },
-    {
-      "epoch": 0.44687777448949395,
-      "grad_norm": 0.55859375,
-      "learning_rate": 7.417133705816837e-05,
-      "loss": 8.0574,
-      "step": 1510
-    },
-    {
-      "epoch": 0.4498372299496893,
-      "grad_norm": 0.63671875,
-      "learning_rate": 7.289574209855559e-05,
-      "loss": 8.0598,
-      "step": 1520
-    },
-    {
-      "epoch": 0.4527966854098846,
-      "grad_norm": 0.66015625,
-      "learning_rate": 7.16248895992645e-05,
-      "loss": 8.0422,
-      "step": 1530
-    },
-    {
-      "epoch": 0.45575614087007993,
-      "grad_norm": 1.046875,
-      "learning_rate": 7.035900192268464e-05,
-      "loss": 7.9814,
-      "step": 1540
-    },
-    {
-      "epoch": 0.45871559633027525,
-      "grad_norm": 1.1796875,
-      "learning_rate": 6.909830056250527e-05,
-      "loss": 7.9703,
-      "step": 1550
-    },
-    {
-      "epoch": 0.4616750517904706,
-      "grad_norm": 0.5390625,
-      "learning_rate": 6.784300610496048e-05,
-      "loss": 8.072,
-      "step": 1560
-    },
-    {
-      "epoch": 0.46463450725066585,
-      "grad_norm": 0.60546875,
-      "learning_rate": 6.65933381902329e-05,
-      "loss": 8.0319,
-      "step": 1570
-    },
-    {
-      "epoch": 0.4675939627108612,
-      "grad_norm": 0.90234375,
-      "learning_rate": 6.534951547402322e-05,
-      "loss": 8.0633,
-      "step": 1580
-    },
-    {
-      "epoch": 0.4705534181710565,
-      "grad_norm": 0.84765625,
-      "learning_rate": 6.411175558929152e-05,
-      "loss": 8.0708,
-      "step": 1590
-    },
-    {
-      "epoch": 0.47351287363125183,
-      "grad_norm": 1.234375,
-      "learning_rate": 6.28802751081779e-05,
-      "loss": 8.0845,
-      "step": 1600
-    },
-    {
-      "epoch": 0.47351287363125183,
-      "eval_loss": 8.029565811157227,
-      "eval_runtime": 11.3209,
-      "eval_samples_per_second": 132.675,
-      "eval_steps_per_second": 33.213,
-      "step": 1600
-    },
-    {
-      "epoch": 0.47647232909144716,
-      "grad_norm": 0.70703125,
-      "learning_rate": 6.165528950410884e-05,
-      "loss": 8.0563,
-      "step": 1610
-    },
-    {
-      "epoch": 0.4794317845516425,
-      "grad_norm": 0.62109375,
-      "learning_rate": 6.0437013114095195e-05,
-      "loss": 7.9905,
-      "step": 1620
-    },
-    {
-      "epoch": 0.4823912400118378,
-      "grad_norm": 0.65625,
-      "learning_rate": 5.922565910122967e-05,
-      "loss": 8.0615,
-      "step": 1630
-    },
-    {
-      "epoch": 0.48535069547203313,
-      "grad_norm": 1.0,
-      "learning_rate": 5.8021439417389444e-05,
-      "loss": 7.8931,
-      "step": 1640
-    },
-    {
-      "epoch": 0.48831015093222846,
-      "grad_norm": 1.3671875,
-      "learning_rate": 5.6824564766150726e-05,
-      "loss": 7.9819,
-      "step": 1650
-    },
-    {
-      "epoch": 0.4912696063924238,
-      "grad_norm": 0.58984375,
-      "learning_rate": 5.563524456592163e-05,
-      "loss": 8.0397,
-      "step": 1660
-    },
-    {
-      "epoch": 0.4942290618526191,
-      "grad_norm": 0.65625,
-      "learning_rate": 5.4453686913300074e-05,
-      "loss": 8.0279,
-      "step": 1670
-    },
-    {
-      "epoch": 0.49718851731281444,
-      "grad_norm": 0.75,
-      "learning_rate": 5.328009854666303e-05,
-      "loss": 8.0236,
-      "step": 1680
-    },
-    {
-      "epoch": 0.5001479727730098,
-      "grad_norm": 1.0625,
-      "learning_rate": 5.2114684809993044e-05,
-      "loss": 8.002,
-      "step": 1690
-    },
-    {
-      "epoch": 0.5031074282332051,
-      "grad_norm": 1.4921875,
-      "learning_rate": 5.095764961694922e-05,
-      "loss": 8.0699,
-      "step": 1700
-    },
-    {
-      "epoch": 0.5060668836934004,
-      "grad_norm": 0.6484375,
-      "learning_rate": 4.980919541518796e-05,
-      "loss": 7.9982,
-      "step": 1710
-    },
-    {
-      "epoch": 0.5090263391535957,
-      "grad_norm": 0.625,
-      "learning_rate": 4.866952315094088e-05,
-      "loss": 8.0741,
-      "step": 1720
-    },
-    {
-      "epoch": 0.511985794613791,
-      "grad_norm": 0.7265625,
-      "learning_rate": 4.753883223385467e-05,
-      "loss": 8.1019,
-      "step": 1730
-    },
-    {
-      "epoch": 0.5149452500739864,
-      "grad_norm": 0.80859375,
-      "learning_rate": 4.6417320502100316e-05,
-      "loss": 8.0329,
-      "step": 1740
-    },
-    {
-      "epoch": 0.5179047055341817,
-      "grad_norm": 1.5234375,
-      "learning_rate": 4.530518418775733e-05,
-      "loss": 8.004,
-      "step": 1750
-    },
-    {
-      "epoch": 0.520864160994377,
-      "grad_norm": 0.625,
-      "learning_rate": 4.4202617882478405e-05,
-      "loss": 8.0818,
-      "step": 1760
-    },
-    {
-      "epoch": 0.5238236164545723,
-      "grad_norm": 0.62109375,
-      "learning_rate": 4.310981450344189e-05,
-      "loss": 7.9803,
-      "step": 1770
-    },
-    {
-      "epoch": 0.5267830719147677,
-      "grad_norm": 0.68359375,
-      "learning_rate": 4.2026965259596666e-05,
-      "loss": 8.0002,
-      "step": 1780
-    },
-    {
-      "epoch": 0.529742527374963,
-      "grad_norm": 1.1640625,
-      "learning_rate": 4.0954259618206295e-05,
-      "loss": 7.9552,
-      "step": 1790
-    },
-    {
-      "epoch": 0.5327019828351583,
-      "grad_norm": 2.703125,
-      "learning_rate": 3.9891885271697496e-05,
-      "loss": 7.9257,
-      "step": 1800
-    },
-    {
-      "epoch": 0.5327019828351583,
-      "eval_loss": 8.028168678283691,
-      "eval_runtime": 11.1134,
-      "eval_samples_per_second": 135.152,
-      "eval_steps_per_second": 33.833,
-      "step": 1800
-    },
-    {
-      "epoch": 0.5356614382953536,
-      "grad_norm": 0.6484375,
-      "learning_rate": 3.884002810481958e-05,
-      "loss": 8.0622,
-      "step": 1810
-    },
-    {
-      "epoch": 0.538620893755549,
-      "grad_norm": 0.640625,
-      "learning_rate": 3.779887216211995e-05,
-      "loss": 8.0267,
-      "step": 1820
-    },
-    {
-      "epoch": 0.5415803492157443,
-      "grad_norm": 0.73046875,
-      "learning_rate": 3.676859961574162e-05,
-      "loss": 7.9916,
-      "step": 1830
-    },
-    {
-      "epoch": 0.5445398046759397,
-      "grad_norm": 0.93359375,
-      "learning_rate": 3.574939073354838e-05,
-      "loss": 8.0598,
-      "step": 1840
-    },
-    {
-      "epoch": 0.5474992601361349,
-      "grad_norm": 1.6171875,
-      "learning_rate": 3.4741423847583134e-05,
-      "loss": 8.0326,
-      "step": 1850
-    },
-    {
-      "epoch": 0.5504587155963303,
-      "grad_norm": 0.59765625,
-      "learning_rate": 3.3744875322865034e-05,
-      "loss": 8.1606,
-      "step": 1860
-    },
-    {
-      "epoch": 0.5534181710565256,
-      "grad_norm": 0.5859375,
-      "learning_rate": 3.275991952653054e-05,
-      "loss": 8.0214,
-      "step": 1870
-    },
-    {
-      "epoch": 0.556377626516721,
-      "grad_norm": 0.73046875,
-      "learning_rate": 3.178672879732435e-05,
-      "loss": 8.0538,
-      "step": 1880
-    },
-    {
-      "epoch": 0.5593370819769162,
-      "grad_norm": 0.9375,
-      "learning_rate": 3.0825473415445074e-05,
-      "loss": 8.0075,
-      "step": 1890
-    },
-    {
-      "epoch": 0.5622965374371116,
-      "grad_norm": 1.8828125,
-      "learning_rate": 2.9876321572751144e-05,
-      "loss": 8.0111,
-      "step": 1900
-    },
-    {
-      "epoch": 0.5652559928973069,
-      "grad_norm": 0.62890625,
-      "learning_rate": 2.8939439343332086e-05,
-      "loss": 8.0155,
-      "step": 1910
-    },
-    {
-      "epoch": 0.5682154483575023,
-      "grad_norm": 0.6875,
-      "learning_rate": 2.8014990654450325e-05,
-      "loss": 7.9791,
-      "step": 1920
-    },
-    {
-      "epoch": 0.5711749038176975,
-      "grad_norm": 0.74609375,
-      "learning_rate": 2.7103137257858868e-05,
-      "loss": 7.9917,
-      "step": 1930
-    },
-    {
-      "epoch": 0.5741343592778929,
-      "grad_norm": 0.796875,
-      "learning_rate": 2.6204038701499056e-05,
-      "loss": 8.0213,
-      "step": 1940
-    },
-    {
-      "epoch": 0.5770938147380882,
-      "grad_norm": 1.6796875,
-      "learning_rate": 2.5317852301584643e-05,
-      "loss": 7.9023,
-      "step": 1950
-    },
-    {
-      "epoch": 0.5800532701982836,
-      "grad_norm": 0.62109375,
-      "learning_rate": 2.4444733115075823e-05,
-      "loss": 8.0239,
-      "step": 1960
-    },
-    {
-      "epoch": 0.5830127256584788,
-      "grad_norm": 0.640625,
-      "learning_rate": 2.3584833912548888e-05,
-      "loss": 7.9794,
-      "step": 1970
-    },
-    {
-      "epoch": 0.5859721811186742,
-      "grad_norm": 0.7109375,
-      "learning_rate": 2.2738305151465645e-05,
-      "loss": 8.009,
-      "step": 1980
-    },
-    {
-      "epoch": 0.5889316365788695,
-      "grad_norm": 0.9453125,
-      "learning_rate": 2.190529494984782e-05,
-      "loss": 8.0134,
-      "step": 1990
-    },
-    {
-      "epoch": 0.5918910920390648,
-      "grad_norm": 1.8203125,
-      "learning_rate": 2.1085949060360654e-05,
-      "loss": 7.8742,
-      "step": 2000
-    },
-    {
-      "epoch": 0.5918910920390648,
-      "eval_loss": 8.027576446533203,
-      "eval_runtime": 11.0208,
-      "eval_samples_per_second": 136.288,
-      "eval_steps_per_second": 34.117,
-      "step": 2000
-    },
-    {
-      "epoch": 0.5948505474992601,
-      "grad_norm": 0.69921875,
-      "learning_rate": 2.0280410844810428e-05,
-      "loss": 8.0312,
-      "step": 2010
-    },
-    {
-      "epoch": 0.5978100029594554,
-      "grad_norm": 0.62890625,
-      "learning_rate": 1.9488821249060297e-05,
-      "loss": 8.0663,
-      "step": 2020
-    },
-    {
-      "epoch": 0.6007694584196508,
-      "grad_norm": 0.7109375,
-      "learning_rate": 1.871131877836879e-05,
-      "loss": 8.0353,
-      "step": 2030
-    },
-    {
-      "epoch": 0.6037289138798461,
-      "grad_norm": 0.80859375,
-      "learning_rate": 1.7948039473155554e-05,
-      "loss": 7.9639,
-      "step": 2040
-    },
-    {
-      "epoch": 0.6066883693400414,
-      "grad_norm": 1.4765625,
-      "learning_rate": 1.7199116885197995e-05,
-      "loss": 8.0724,
-      "step": 2050
-    },
-    {
-      "epoch": 0.6096478248002367,
-      "grad_norm": 0.6953125,
-      "learning_rate": 1.646468205426377e-05,
-      "loss": 8.0524,
-      "step": 2060
-    },
-    {
-      "epoch": 0.6126072802604321,
-      "grad_norm": 0.59375,
-      "learning_rate": 1.5744863485182537e-05,
-      "loss": 8.0682,
-      "step": 2070
-    },
-    {
-      "epoch": 0.6155667357206274,
-      "grad_norm": 0.71484375,
-      "learning_rate": 1.5039787125361326e-05,
-      "loss": 8.0506,
-      "step": 2080
-    },
-    {
-      "epoch": 0.6185261911808227,
-      "grad_norm": 1.015625,
-      "learning_rate": 1.4349576342747462e-05,
-      "loss": 8.0654,
-      "step": 2090
-    },
-    {
-      "epoch": 0.621485646641018,
-      "grad_norm": 1.6875,
-      "learning_rate": 1.3674351904242611e-05,
-      "loss": 7.922,
-      "step": 2100
-    },
-    {
-      "epoch": 0.6244451021012134,
-      "grad_norm": 0.625,
-      "learning_rate": 1.3014231954572287e-05,
-      "loss": 8.0273,
-      "step": 2110
-    },
-    {
-      "epoch": 0.6274045575614087,
-      "grad_norm": 0.8203125,
-      "learning_rate": 1.2369331995613665e-05,
-      "loss": 8.0164,
-      "step": 2120
-    },
-    {
-      "epoch": 0.630364013021604,
-      "grad_norm": 0.76953125,
-      "learning_rate": 1.173976486618631e-05,
-      "loss": 8.0,
-      "step": 2130
-    },
-    {
-      "epoch": 0.6333234684817993,
-      "grad_norm": 0.95703125,
-      "learning_rate": 1.1125640722308628e-05,
-      "loss": 8.1147,
-      "step": 2140
-    },
-    {
-      "epoch": 0.6362829239419947,
-      "grad_norm": 1.578125,
-      "learning_rate": 1.0527067017923654e-05,
-      "loss": 7.8947,
-      "step": 2150
-    },
-    {
-      "epoch": 0.63924237940219,
-      "grad_norm": 0.58984375,
-      "learning_rate": 9.944148486097793e-06,
-      "loss": 8.0644,
-      "step": 2160
-    },
-    {
-      "epoch": 0.6422018348623854,
-      "grad_norm": 0.59765625,
-      "learning_rate": 9.376987120695545e-06,
-      "loss": 8.0381,
-      "step": 2170
-    },
-    {
-      "epoch": 0.6451612903225806,
-      "grad_norm": 0.75,
-      "learning_rate": 8.825682158533554e-06,
-      "loss": 7.944,
-      "step": 2180
-    },
-    {
-      "epoch": 0.648120745782776,
-      "grad_norm": 0.94140625,
-      "learning_rate": 8.290330062017016e-06,
-      "loss": 8.0418,
-      "step": 2190
-    },
-    {
-      "epoch": 0.6510802012429713,
-      "grad_norm": 1.3359375,
-      "learning_rate": 7.771024502261526e-06,
-      "loss": 8.0265,
-      "step": 2200
-    },
-    {
-      "epoch": 0.6510802012429713,
-      "eval_loss": 8.027557373046875,
-      "eval_runtime": 11.1335,
-      "eval_samples_per_second": 134.908,
-      "eval_steps_per_second": 33.772,
-      "step": 2200
-    },
-    {
-      "epoch": 0.6540396567031667,
-      "grad_norm": 0.5703125,
-      "learning_rate": 7.267856342703461e-06,
-      "loss": 8.093,
-      "step": 2210
-    },
-    {
-      "epoch": 0.6569991121633619,
-      "grad_norm": 0.69140625,
-      "learning_rate": 6.780913623201346e-06,
-      "loss": 7.9956,
-      "step": 2220
-    },
-    {
-      "epoch": 0.6599585676235573,
-      "grad_norm": 0.8125,
-      "learning_rate": 6.310281544631546e-06,
-      "loss": 8.0602,
-      "step": 2230
-    },
-    {
-      "epoch": 0.6629180230837526,
-      "grad_norm": 0.83203125,
-      "learning_rate": 5.856042453980526e-06,
-      "loss": 8.0176,
-      "step": 2240
-    },
-    {
-      "epoch": 0.665877478543948,
-      "grad_norm": 1.78125,
-      "learning_rate": 5.418275829936537e-06,
-      "loss": 7.899,
-      "step": 2250
-    },
-    {
-      "epoch": 0.6688369340041432,
-      "grad_norm": 0.6015625,
-      "learning_rate": 4.997058268983135e-06,
-      "loss": 8.1306,
-      "step": 2260
-    },
-    {
-      "epoch": 0.6717963894643386,
-      "grad_norm": 0.6875,
-      "learning_rate": 4.592463471997022e-06,
-      "loss": 8.0341,
-      "step": 2270
-    },
-    {
-      "epoch": 0.6747558449245339,
-      "grad_norm": 0.62109375,
-      "learning_rate": 4.204562231352516e-06,
-      "loss": 8.0087,
-      "step": 2280
-    },
-    {
-      "epoch": 0.6777153003847292,
-      "grad_norm": 0.765625,
-      "learning_rate": 3.83342241853496e-06,
-      "loss": 8.0114,
-      "step": 2290
-    },
-    {
-      "epoch": 0.6806747558449245,
-      "grad_norm": 1.625,
-      "learning_rate": 3.4791089722651436e-06,
-      "loss": 7.88,
-      "step": 2300
-    },
-    {
-      "epoch": 0.6836342113051198,
-      "grad_norm": 0.72265625,
-      "learning_rate": 3.1416838871368924e-06,
-      "loss": 7.9878,
-      "step": 2310
-    },
-    {
-      "epoch": 0.6865936667653152,
-      "grad_norm": 0.6328125,
-      "learning_rate": 2.821206202769899e-06,
-      "loss": 8.0762,
-      "step": 2320
-    },
-    {
-      "epoch": 0.6895531222255105,
-      "grad_norm": 1.046875,
-      "learning_rate": 2.5177319934794e-06,
-      "loss": 7.9838,
-      "step": 2330
-    },
-    {
-      "epoch": 0.6925125776857058,
-      "grad_norm": 0.87109375,
-      "learning_rate": 2.2313143584648423e-06,
-      "loss": 8.0461,
-      "step": 2340
-    },
-    {
-      "epoch": 0.6954720331459011,
-      "grad_norm": 1.515625,
-      "learning_rate": 1.9620034125190644e-06,
-      "loss": 8.1426,
-      "step": 2350
-    },
-    {
-      "epoch": 0.6984314886060965,
-      "grad_norm": 0.55859375,
-      "learning_rate": 1.7098462772596302e-06,
-      "loss": 7.9893,
-      "step": 2360
-    },
-    {
-      "epoch": 0.7013909440662918,
-      "grad_norm": 0.66015625,
-      "learning_rate": 1.4748870728839347e-06,
-      "loss": 8.0056,
-      "step": 2370
-    },
-    {
-      "epoch": 0.7043503995264871,
-      "grad_norm": 0.58984375,
-      "learning_rate": 1.2571669104494256e-06,
-      "loss": 8.0057,
-      "step": 2380
-    },
-    {
-      "epoch": 0.7073098549866824,
-      "grad_norm": 0.9765625,
-      "learning_rate": 1.0567238846803996e-06,
-      "loss": 7.9673,
-      "step": 2390
-    },
-    {
-      "epoch": 0.7102693104468778,
-      "grad_norm": 1.03125,
-      "learning_rate": 8.735930673024806e-07,
-      "loss": 7.8113,
-      "step": 2400
-    },
-    {
-      "epoch": 0.7102693104468778,
-      "eval_loss": 8.03307819366455,
-      "eval_runtime": 11.143,
-      "eval_samples_per_second": 134.794,
-      "eval_steps_per_second": 33.743,
-      "step": 2400
-    },
-    {
-      "epoch": 0.7132287659070731,
-      "grad_norm": 0.6171875,
-      "learning_rate": 7.078065009060941e-07,
-      "loss": 7.9873,
-      "step": 2410
-    },
-    {
-      "epoch": 0.7161882213672685,
-      "grad_norm": 0.62109375,
-      "learning_rate": 5.593931933399854e-07,
-      "loss": 8.0612,
-      "step": 2420
-    },
-    {
-      "epoch": 0.7191476768274637,
-      "grad_norm": 0.69140625,
-      "learning_rate": 4.2837911263562404e-07,
-      "loss": 8.0285,
-      "step": 2430
-    },
-    {
-      "epoch": 0.7221071322876591,
-      "grad_norm": 0.921875,
-      "learning_rate": 3.1478718246357173e-07,
-      "loss": 8.0167,
-      "step": 2440
-    },
-    {
-      "epoch": 0.7250665877478544,
-      "grad_norm": 1.8203125,
-      "learning_rate": 2.1863727812254653e-07,
-      "loss": 8.1032,
-      "step": 2450
-    },
-    {
-      "epoch": 0.7280260432080498,
-      "grad_norm": 0.5390625,
-      "learning_rate": 1.3994622306173765e-07,
-      "loss": 8.0447,
-      "step": 2460
-    },
-    {
-      "epoch": 0.730985498668245,
-      "grad_norm": 0.609375,
-      "learning_rate": 7.872778593728258e-08,
-      "loss": 8.0293,
-      "step": 2470
-    },
-    {
-      "epoch": 0.7339449541284404,
-      "grad_norm": 0.78125,
-      "learning_rate": 3.499267820307184e-08,
-      "loss": 8.0606,
-      "step": 2480
-    },
-    {
-      "epoch": 0.7369044095886357,
-      "grad_norm": 0.8515625,
-      "learning_rate": 8.748552236603757e-09,
-      "loss": 8.0531,
-      "step": 2490
-    },
-    {
-      "epoch": 0.7398638650488311,
-      "grad_norm": 1.265625,
-      "learning_rate": 0.0,
-      "loss": 7.9245,
-      "step": 2500
     }
   ],
   "logging_steps": 10,
   "max_steps": 2500,
   "num_input_tokens_seen": 0,
-  "num_train_epochs": 1,
-  "save_steps": 400,
+  "num_train_epochs": 49,
+  "save_steps": 40,
   "stateful_callbacks": {
     "TrainerControl": {
       "args": {
@@ -1875,13 +57,13 @@
         "should_evaluate": false,
         "should_log": false,
         "should_save": true,
-        "should_training_stop": true
+        "should_training_stop": false
       },
       "attributes": {}
     }
   },
-  "total_flos": 128870887981056.0,
-  "train_batch_size": 4,
+  "total_flos": 6036802020311040.0,
+  "train_batch_size": 1,
   "trial_name": null,
   "trial_params": null
 }

last-checkpoint/training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8ffd459aa83253f73e371e129574dd0434ee79c2c18eb103b0e4428a34062eb2
+oid sha256:ab35bb512bd0815bfb48651f4d2ed30517d244444ffa68a91ce2ff049faa81b0
 size 6840