File size: 1,766 Bytes
3124298 ea7303f 03f490b ea7303f 3124298 ea7303f 3124298 ea7303f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 |
---
language: en
license: mit
tags:
- spiking-neural-networks
- language-modeling
- neuromorphic
- energy-efficient
- biological-ai
datasets:
- fineweb-5B
pipeline_tag: text-generation
---
# 🧠 Spiking Neural Network Language Model - Training Checkpoint
**Live training checkpoint from the world's first large-scale spiking language model!**
## Current Training Status
- **Training Step**: 554,000
- **Tokens Processed**: 5.67B tokens
- **Current Loss**: 4.5783
- **Spike Rate**: 0.0508
- **Learning Rate**: 8.15e-05
## Model Architecture
- **Parameters**: ~54M
- **Architecture**: 12-layer Spiking LTC Network
- **Hidden Size**: 768
- **Sequence Length**: 1024
- **Multi-timescale Processing**: Fast → Medium → Slow layers
## Training Details
- **Dataset**: PatrickHaller/fineweb-5B
- **Target**: 3 epochs (~15B tokens total)
- **Biological Dynamics**: Adaptive thresholds, refractory periods
- **Energy Efficiency**: ~5% neuron activation vs 100% in Transformers
## Scientific Significance
This represents ongoing training of the first large-scale spiking neural network for language modeling, demonstrating:
1. **Biological neural dynamics** can learn language at scale
2. **Energy efficiency** through sparse neural firing
3. **Multi-timescale processing** for hierarchical understanding
## Usage
```python
# Download this checkpoint
from huggingface_hub import hf_hub_download
checkpoint = hf_hub_download(
repo_id="rootxhacker/piking-llm-5b-3epochs-exp",
filename="checkpoint_554000.pt"
)
# Load with custom spiking model code
# (See full implementation for complete usage)
```
---
**🔬 This is live research in progress! Check back for updates as training continues.**
**Training Progress**: 37.8% complete towards 15B tokens
|