Training checkpoint 4000 - 0.0B tokens

README.md

@@ -18,11 +18,11 @@ pipeline_tag: text-generation
 
 ## Current Training Status
 
-- **Training Step**: 2,000
-- **Tokens Processed**: 0.02B tokens
-- **Current Loss**: 9.8659
-- **Spike Rate**: 0.0115
-- **Learning Rate**: 1.02e-05
+- **Training Step**: 4,000
+- **Tokens Processed**: 0.04B tokens
+- **Current Loss**: 5.8477
+- **Spike Rate**: 0.0111
+- **Learning Rate**: 1.67e-05
 
 ## Model Architecture
 
@@ -54,7 +54,7 @@ This represents ongoing training of the first large-scale spiking neural network
 from huggingface_hub import hf_hub_download
 checkpoint = hf_hub_download(
     repo_id="rootxhacker/piking-llm-5b-3epochs-exp",
-    filename="checkpoint_2000.pt"
+    filename="checkpoint_4000.pt"
 )
 
 # Load with custom spiking model code
@@ -65,4 +65,4 @@ checkpoint = hf_hub_download(
 
 **🔬 This is live research in progress! Check back for updates as training continues.**
 
-**Training Progress**: 0.1% complete towards 15B tokens
+**Training Progress**: 0.3% complete towards 15B tokens

checkpoint_4000.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e48449a2723144db8c84ea4ee33be104e5197d4c46360e34e847e8f51f3ecb97
+size 999026318

config.json

@@ -4,11 +4,11 @@
   "hidden_size": 768,
   "num_layers": 12,
   "max_seq_length": 1024,
-  "training_step": 2000,
-  "tokens_processed": 20480000,
-  "loss": 9.865856721571749,
-  "spike_rate": 0.011451726761236444,
-  "learning_rate": 1.019965406348427e-05,
-  "epoch": 0.004096,
-  "progress_percent": 0.13653368285956147
+  "training_step": 4000,
+  "tokens_processed": 40960000,
+  "loss": 5.847672875415472,
+  "spike_rate": 0.011124921468187653,
+  "learning_rate": 1.669781462895783e-05,
+  "epoch": 0.008192,
+  "progress_percent": 0.27306736571912293
 }