Training checkpoint 48000 - 0.5B tokens

README.md

@@ -18,10 +18,10 @@ pipeline_tag: text-generation
 
 ## Current Training Status
 
-- **Training Step**: 46,000
-- **Tokens Processed**: 0.47B tokens
-- **Current Loss**: 4.6367
-- **Spike Rate**: 0.0466
+- **Training Step**: 48,000
+- **Tokens Processed**: 0.49B tokens
+- **Current Loss**: 4.7083
+- **Spike Rate**: 0.0489
 - **Learning Rate**: 2.00e-04
 
 ## Model Architecture
@@ -54,7 +54,7 @@ This represents ongoing training of the first large-scale spiking neural network
 from huggingface_hub import hf_hub_download
 checkpoint = hf_hub_download(
     repo_id="rootxhacker/piking-llm-5b-3epochs-exp",
-    filename="checkpoint_46000.pt"
+    filename="checkpoint_48000.pt"
 )
 
 # Load with custom spiking model code
@@ -65,4 +65,4 @@ checkpoint = hf_hub_download(
 
 **🔬 This is live research in progress! Check back for updates as training continues.**
 
-**Training Progress**: 3.1% complete towards 15B tokens
+**Training Progress**: 3.3% complete towards 15B tokens

checkpoint_48000.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07996b07b3e4759f1e526f1de4b8b3527203bfdba07c7ad5a07409c83c57a509
+size 999026492

config.json

@@ -4,11 +4,11 @@
   "hidden_size": 768,
   "num_layers": 12,
   "max_seq_length": 1024,
-  "training_step": 46000,
-  "tokens_processed": 471040000,
-  "loss": 4.6367024988494165,
-  "spike_rate": 0.046578304323731834,
-  "learning_rate": 0.00019993323564906977,
-  "epoch": 0.094208,
-  "progress_percent": 3.1402747057699134
+  "training_step": 48000,
+  "tokens_processed": 491520000,
+  "loss": 4.708328228632757,
+  "spike_rate": 0.04892981921125185,
+  "learning_rate": 0.0001999162945551328,
+  "epoch": 0.098304,
+  "progress_percent": 3.2768083886294748
 }