Training checkpoint 2000 - 0.0B tokens

README.md

@@ -0,0 +1,68 @@
+---
+language: en
+license: mit
+tags:
+- spiking-neural-networks
+- language-modeling
+- neuromorphic
+- energy-efficient
+- biological-ai
+datasets:
+- fineweb-5B
+pipeline_tag: text-generation
+---
+
+# 🧠 Spiking Neural Network Language Model - Training Checkpoint
+
+**Live training checkpoint from the world's first large-scale spiking language model!**
+
+## Current Training Status
+
+- **Training Step**: 2,000
+- **Tokens Processed**: 0.02B tokens
+- **Current Loss**: 9.8659
+- **Spike Rate**: 0.0115
+- **Learning Rate**: 1.02e-05
+
+## Model Architecture
+
+- **Parameters**: ~54M
+- **Architecture**: 12-layer Spiking LTC Network
+- **Hidden Size**: 768
+- **Sequence Length**: 1024
+- **Multi-timescale Processing**: Fast → Medium → Slow layers
+
+## Training Details
+
+- **Dataset**: PatrickHaller/fineweb-5B
+- **Target**: 3 epochs (~15B tokens total)
+- **Biological Dynamics**: Adaptive thresholds, refractory periods
+- **Energy Efficiency**: ~5% neuron activation vs 100% in Transformers
+
+## Scientific Significance
+
+This represents ongoing training of the first large-scale spiking neural network for language modeling, demonstrating:
+
+1. **Biological neural dynamics** can learn language at scale
+2. **Energy efficiency** through sparse neural firing
+3. **Multi-timescale processing** for hierarchical understanding
+
+## Usage
+
+```python
+# Download this checkpoint
+from huggingface_hub import hf_hub_download
+checkpoint = hf_hub_download(
+    repo_id="rootxhacker/piking-llm-5b-3epochs-exp",
+    filename="checkpoint_2000.pt"
+)
+
+# Load with custom spiking model code
+# (See full implementation for complete usage)
+```
+
+---
+
+**🔬 This is live research in progress! Check back for updates as training continues.**
+
+**Training Progress**: 0.1% complete towards 15B tokens

checkpoint_2000.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2525c357147a84397048617fc112cfaebe7389e44328d60cac9eb224f13894d
+size 999026318

config.json

@@ -0,0 +1,14 @@
+{
+  "model_type": "spiking_llm",
+  "vocab_size": 50257,
+  "hidden_size": 768,
+  "num_layers": 12,
+  "max_seq_length": 1024,
+  "training_step": 2000,
+  "tokens_processed": 20480000,
+  "loss": 9.865856721571749,
+  "spike_rate": 0.011451726761236444,
+  "learning_rate": 1.019965406348427e-05,
+  "epoch": 0.004096,
+  "progress_percent": 0.13653368285956147
+}