CLAUDE.md

# BitTransformerLM Claude Code Integration Guide

## Overview

BitTransformerLM is optimally designed for use with [Claude Code](https://claude.ai/code), providing AI-assisted setup, development, and research workflows. This document provides guidelines for working with BitTransformerLM in Claude Code and standalone development.

## Why Claude Code?

BitTransformerLM's unique bit-native architecture has several complexities that Claude Code can help navigate:

- **Complex Architecture**: Understanding bit-level processing, reversible layers, and safety telemetry
- **Parameter Tuning**: Optimizing model configurations for different use cases
- **Safety Monitoring**: Interpreting K/C/S metrics and configuring safety gates
- **Distributed Training**: Setting up FSDP and pipeline parallelism correctly
- **Debugging**: Identifying issues specific to bit-native processing

Claude Code understands these nuances and can provide real-time assistance.

---

## Repository Scope and Architecture

### Core Capabilities
BitTransformerLM implements bit-native language modeling with:
- **Bit-Native Processing**: Direct binary sequence modeling with parity protection
- **Reversible Layers**: Memory-efficient transformer blocks that save ~50% memory
- **Safety Telemetry**: Real-time K/C/S (Negentropy/Complexity/Symbiosis) monitoring
- **Diffusion Mode**: Bidirectional denoising with multiple noise schedules
- **Progressive Scaling**: Automatic model expansion based on validation performance
- **Distributed Training**: FSDP and pipeline parallelism for large-scale training
- **Interactive Dashboard**: Real-time training control and visualization

### Experimental Status
**Important**: BitTransformerLM is experimental research software requiring:
- Rigorous baseline comparisons against standard transformers
- Validation on established language modeling benchmarks
- Statistical significance testing across multiple runs
- Careful interpretation of safety metrics and claims

---

## Environment Setup

### Requirements
- **Python 3.10+** (required for modern PyTorch features)
- **PyTorch 2.7.1+** with appropriate CUDA support if using GPUs

### Installation Options

#### CPU-Only Installation
```bash
pip install --extra-index-url https://download.pytorch.org/whl/cpu -r requirements.txt
```

#### GPU Installation  
```bash
pip install --extra-index-url https://download.pytorch.org/whl/cu118 torch==2.7.1+cu118
pip install -r requirements.txt
```

#### Claude Code Assisted Setup
When using Claude Code, simply ask for:
- "Help me set up BitTransformerLM for my system"
- "Configure BitTransformerLM for GPU training"
- "Set up a development environment for bit-native language modeling"

Claude Code will guide you through hardware detection, dependency installation, and initial configuration.

---

## Repository Structure

```
BitTransformerLM/
├── bit_transformer/              # Core package
│   ├── model.py                  # BitTransformerLM architecture
│   ├── telemetry.py              # K/C/S safety metrics
│   ├── safety.py                 # Safety gates and monitoring
│   ├── bit_io.py                 # Text ↔ bits conversion
│   ├── compression.py            # Run-length encoding
│   ├── training.py               # Training utilities
│   ├── distributed.py            # FSDP and pipeline parallelism
│   ├── dashboard_app.py          # Interactive web dashboard
│   ├── quantization.py           # INT8/4-bit quantization
│   └── [other modules...]        # Additional functionality
├── tests/                        # Test suite and results
├── example.py                    # Basic usage example
├── unified_workflow.py           # Main training script
├── mcp_server.py                 # Management Control Protocol server
├── USER_GUIDE.md                 # Comprehensive user documentation
└── [other scripts...]            # Utilities and examples
```

---

## Development Workflow with Claude Code

### Getting Started

1. **Initial Setup**
   ```
   "Help me understand BitTransformerLM's architecture"
   "Create a simple training script for bit-native language modeling"
   "Explain the difference between causal and diffusion modes"
   ```

2. **Model Configuration**
   ```
   "Configure a BitTransformerLM for [my specific use case]"
   "What are optimal hyperparameters for a [size] model?"
   "Help me enable reversible layers and gradient checkpointing"
   ```

3. **Training and Monitoring**
   ```
   "Set up distributed training with FSDP"
   "Interpret these K/C/S telemetry values: K=0.3, C=0.6, S=0.4"
   "Debug this memory error during training"
   ```

### Claude Code Advantages

**Real-time Assistance**: Get immediate help with:
- Parameter configuration and tuning
- Error diagnosis and resolution  
- Architecture modification and experimentation
- Safety metric interpretation
- Performance optimization

**Context-Aware Suggestions**: Claude Code understands:
- BitTransformerLM's unique bit-native processing
- The relationship between safety metrics
- Memory optimization strategies
- Distributed training complexities

---

## Key Commands and Workflows

### Basic Usage
```bash
# Run simple example
python example.py

# Launch interactive dashboard
python unified_workflow.py --dashboard

# Train with diffusion mode
python unified_workflow.py --diffusion --diffusion-steps 8 --dataset-size 32
```

### Advanced Training
```bash
# Distributed training with FSDP
python unified_workflow.py --distributed --batch-size 2 --epochs 10

# Mixed precision with quantization
python unified_workflow.py --amp --qat

# Progressive scaling with curriculum learning
python unified_workflow.py --progressive --diffusion-curriculum
```

### Dashboard and Monitoring
```bash
# Start MCP server and dashboard
python mcp_server.py &
MCP_SERVER_ADDR=http://127.0.0.1:7000 python -m bit_transformer.dashboard_app
```

**Dashboard Features:**
- Real-time telemetry visualization
- Interactive model configuration
- HuggingFace checkpoint management
- Safe inference testing interface

---

## Safety and Telemetry

### Core Metrics

| Metric | Full Name | Range | Interpretation |
|--------|-----------|-------|----------------|
| **K** | Negentropy | 0-1 | Information content (0=noise, 1=ordered) |
| **C** | LZ Complexity | 0-1 | Pattern complexity (higher=more complex) |
| **S** | Symbiosis | 0-1 | Alignment with reference (higher=better) |

### Using with Claude Code

```
"Explain what K=0.2, C=0.8, S=0.3 means for my model"
"Configure safety gates for production use"  
"My model is generating repetitive output, what safety metrics should I check?"
"Set up drift detection for telemetry monitoring"
```

Claude Code can help interpret these metrics in context and suggest appropriate safety thresholds.

### Safety Gate Configuration
```python
from bit_transformer.safety import SafetyGate

# Production-ready safety gate
gate = SafetyGate(
    c_floor=0.3,      # Minimum complexity
    s_floor=0.5,      # Minimum symbiosis
    decay=0.9,        # EMA decay factor
    burn_in=10        # Steps before gating starts
)
```

---

## Best Practices for Claude Code Development

### 1. **Always Validate Research Claims**
Ask Claude Code to help you:
- Set up proper baseline comparisons
- Design statistical significance tests
- Implement evaluation on standard benchmarks
- Document limitations and assumptions

### 2. **Use Progressive Development**
```
"Start me with a minimal BitTransformerLM example"
"Now add safety monitoring"
"Scale up to distributed training"
"Add diffusion mode capabilities"
```

### 3. **Leverage Claude Code for Architecture Understanding**
```
"Explain how reversible layers save memory"
"Walk me through the bit encoding process"
"How does the safety telemetry system work?"
"Compare BitTransformerLM to standard transformers"
```

### 4. **Get Help with Complex Configurations**
```python
# Ask Claude Code to help configure models like:
model = BitTransformerLM(
    d_model=1024,           # Claude Code can suggest optimal values
    nhead=16,               # Based on your hardware and use case
    num_layers=20,
    dim_feedforward=4096,
    max_seq_len=2048,
    reversible=True,        # Memory optimization
    use_checkpoint=True,    # Gradient checkpointing
    chunk_size=256,         # Attention chunking
    lambda_K=0.1,           # Regularization weights
    lambda_C=0.1,
    lambda_S=0.1
)
```

---

## Development Guidelines

### Code Style
- **Functions**: `snake_case` (e.g., `train_loop`, `safe_inference`)
- **Classes**: `CamelCase` (e.g., `BitTransformerLM`, `SafetyGate`)
- **Constants**: `UPPER_SNAKE_CASE` (e.g., `MAX_SEQ_LEN`)
- **Keep functions under 300 lines** and minimize deep nesting

### Security and Safety
- **Never reintroduce deprecated `/exec` endpoint**
- **Always use safety gates in production**
- **Validate all user inputs** in dashboard and API endpoints
- **Monitor telemetry metrics** for anomalous behavior
- **Use `cpu_autocast()` helper** instead of direct `torch.amp.autocast`

### Memory Management
```python
# Good: Memory-efficient configuration
model = BitTransformerLM(
    reversible=True,        # Enable reversible layers
    use_checkpoint=True,    # Gradient checkpointing  
    chunk_size=128,         # Chunked attention
    full_attn_logging=False # Skip full attention reconstruction
)

# Training with memory optimizations
train_loop(
    model, data,
    amp=True,              # Mixed precision
    accum_steps=4,         # Gradient accumulation
    compile_model=True     # torch.compile optimization
)
```

### Testing and Validation
```bash
# Run tests after changes
pytest -q

# Model evaluation modes
model.train()    # For training
model.eval()     # For inference/evaluation
set_dropout(model, 0.0)  # Disable dropout for reproducible results
```

---

## Getting Help from Claude Code

### Specific Areas Where Claude Code Excels

1. **Architecture Design**
   - "Design a BitTransformerLM architecture for [specific task]"
   - "Optimize this model configuration for memory efficiency"
   - "Explain the trade-offs between reversible and standard layers"

2. **Training Optimization**
   - "My training is running out of memory, help optimize"
   - "Configure distributed training for 4 GPUs"
   - "Set up a training curriculum for bit-native language modeling"

3. **Safety and Monitoring**
   - "Interpret these telemetry readings and suggest adjustments"
   - "Set up production-ready safety monitoring"
   - "Debug why my safety gate is triggering constantly"

4. **Research and Evaluation**
   - "Design a rigorous evaluation comparing BitTransformerLM to GPT-2"
   - "Set up statistical significance testing for my experiments"
   - "Help me write up my research findings honestly"

### Sample Claude Code Interactions

```
User: "I'm getting OOM errors training a 500M parameter BitTransformerLM"

Claude Code: "Let me help optimize your memory usage. Here's a configuration 
that should work better for your model size:

model = BitTransformerLM(
    d_model=768,
    nhead=12, 
    num_layers=12,
    reversible=True,          # Critical for large models
    use_checkpoint=True,      # Trade compute for memory
    chunk_size=64,            # Reduce attention memory
    full_attn_logging=False   # Skip expensive logging
)

train_loop(
    model, data,
    batch_size=1,             # Small batch size
    accum_steps=16,           # Maintain effective batch size
    amp=True                  # Mixed precision training
)

This should reduce memory usage by ~60% compared to standard configuration."
```

---

## Licensing and Distribution

BitTransformerLM is available under dual licensing:
- **Open Source**: AGPLv3 for research and open source use
- **Commercial**: Contact **[email protected]** for commercial licensing

When working with Claude Code, ensure compliance with the AGPLv3 license for any derivatives or modifications you create.

---

## Research Integrity

**Important Reminder**: BitTransformerLM is experimental research software. When using Claude Code:

1. **Always validate claims** through proper baseline comparisons
2. **Document limitations** honestly in any publications or reports  
3. **Use statistical significance testing** for any performance claims
4. **Follow established ML research best practices**
5. **Share negative results** as well as positive ones

Claude Code can help you design rigorous experiments and avoid common pitfalls in ML research.

---

## Support and Community

### Getting Help
- **Claude Code**: Real-time AI assistance with BitTransformerLM
- **GitHub Issues**: Bug reports and feature requests
- **Discussions**: Community questions and sharing
- **User Guide**: Comprehensive documentation (`USER_GUIDE.md`)
- **Project Overview**: Complete project information (`ABOUTME.md`)

### Contributing
When contributing to BitTransformerLM:
1. Use Claude Code to ensure code quality and consistency
2. Follow the development guidelines in this document
3. Add tests for new functionality
4. Update documentation as needed
5. Ensure all safety and security practices are followed

---

**BitTransformerLM + Claude Code provides a powerful combination for exploring bit-native language modeling with AI assistance. Start experimenting responsibly and share your findings with the research community!** 🤖✨