# Temporal Difference Learning & Q-Learning Implementation

A comprehensive implementation of Temporal Difference Learning algorithms featuring TD(0), detailed educational content, and practical reinforcement learning applications with extensive logging and visualization capabilities.

## 📋 Project Overview

This project provides a complete learning experience for Temporal Difference Learning, one of the most fundamental algorithms in reinforcement learning. It demonstrates how agents can learn state values by bootstrapping from current estimates rather than waiting for complete episodes, making it more efficient than Monte Carlo methods.

## 🎯 Key Features

- **Educational Content**: Comprehensive learning materials with step-by-step explanations
- **Complete TD(0) Implementation**: Core temporal difference learning algorithm
- **Detailed Logging**: Every TD update tracked and logged for analysis
- **Real-time Visualization**: Value function evolution and convergence plots
- **Comprehensive Metrics**: Training progress, TD errors, and convergence analysis
- **Auto-save Results**: JSON export of all training data and paramettaers
- **Cross-platform Compatible**: Works on Apple Silicon, Intel, and Google Colab
- **Performance Analysis**: Detailed convergence studies and hyperparameter effects

## 📁 Project Structure

```
├── TLearningRL.ipynb                    # Main notebook with theory and implementation
├── readme.md                            # This file
├── Study Mode - Temporal Difference Learning.pdf  # Educational PDF guide
├── td_learning_20250802_094606.json    # Training results and metrics
└── td_learning_plots_20250802_094606.png # Visualization outputs
```

## 🚀 Getting Started

### Prerequisites
```bash
pip install numpy matplotlib pandas seaborn jupyter
```

### Running the Project
1. Open `TLearningRL.ipynb` in Jupyter Notebook
2. Run all cells to see the complete learning experience
3. The notebook includes:
   - Theoretical explanations with real-life analogies
   - Step-by-step TD learning implementation
   - Interactive visualizations and convergence analysis
   - Performance metrics and practical applications

## 🧮 Algorithm Implementation

### TD(0) Learning
- **Method**: Temporal Difference learning with 0-step lookahead
- **Update Rule**: V(s) ← V(s) + α[r + γV(s') - V(s)]
- **Key Advantage**: Online learning without waiting for episode completion
- **Application**: State value function estimation

### Key Parameters
- **Alpha (α)**: Learning rate (0.1) - controls update speed
- **Gamma (γ)**: Discount factor (0.9) - importance of future rewards
- **Episodes**: Training iterations (100) - total learning experiences

## 📊 Key Results

### Final State Values
- **State 0**: 2.42 (starting position)
- **State 1**: 4.85 (intermediate state)
- **State 2**: 6.91 (closer to goal)
- **State 3**: 8.67 (near terminal state)
- **State 4**: 0.00 (terminal state)

### Training Metrics
- **Convergence**: Achieved within 100 episodes
- **TD Error Reduction**: From 2.0+ to <1.5
- **Value Propagation**: Backward from terminal state
- **Learning Efficiency**: Online updates every step

## 🧠 Learning Content

The notebook includes comprehensive educational material:

1. **TD Learning Fundamentals** - Bootstrapping and online learning concepts
2. **Algorithm Mechanics** - Step-by-step TD update process
3. **Value Function Evolution** - How state values propagate and converge
4. **Convergence Analysis** - Understanding TD error reduction patterns
5. **Hyperparameter Effects** - Impact of learning rate and discount factor
6. **Practical Applications** - Real-world uses in AI and robotics

## 🔍 Key Concepts Covered

- **Temporal Difference Error**: The "surprise" signal that drives learning
- **Bootstrapping**: Using current estimates to improve future estimates
- **Online Learning**: Immediate updates vs. batch processing
- **Value Function Convergence**: How estimates improve over time
- **Exploration vs. Exploitation**: Balancing learning and performance

## 📈 Visualizations

- **Value Function Evolution**: State values over training episodes
- **TD Error Convergence**: Learning progress and stability
- **Training Progression**: Episode rewards and performance metrics
- **Parameter Sensitivity**: Effects of different hyperparameter settings

## 🎓 Educational Value

This project serves as a complete learning resource for understanding Temporal Difference Learning, combining:

- **Theoretical Foundation**: Mathematical principles with intuitive explanations
- **Practical Implementation**: Working code with detailed logging
- **Visual Learning**: Interactive plots showing algorithm behavior
- **Performance Analysis**: Understanding convergence and stability
- **Real-world Context**: Applications in modern AI systems

Perfect for:
- Students learning reinforcement learning fundamentals
- Researchers implementing TD-based algorithms
- Practitioners building adaptive AI systems
- Anyone interested in online learning algorithms

## 🔬 Real-World Applications

- **Game AI**: Learning game positions and strategies (chess, Go)
- **Robotics**: Adaptive control and navigation systems
- **Finance**: Real-time trading strategy optimization
- **Recommendation Systems**: Online preference learning
- **Autonomous Vehicles**: Dynamic route and behavior optimization
- **Resource Management**: Adaptive scheduling and allocation

## 📈 Output Files

### Automatic Saves
- `td_learning_YYYYMMDD_HHMMSS.json` - Complete training data
- `td_learning_plots_YYYYMMDD_HHMMSS.png` - Visualization plots

### JSON Structure
```json
{
  "parameters": {
    "alpha": 0.1,
    "gamma": 0.9,
    "num_states": 5
  },
  "final_values": [2.42, 4.85, 6.91, 8.67, 0.0],
  "training_metrics": {
    "episodes": [...],
    "total_rewards": [...],
    "avg_td_error": [...]
  }
}
```

## 🧠 Algorithm Details

### TD(0) Update Rule
```
V(s) ← V(s) + α[r + γV(s') - V(s)]
```

Where:
- `V(s)`: Current state value estimate
- `α`: Learning rate
- `r`: Immediate reward
- `γ`: Discount factor
- `V(s')`: Next state value estimate

### Key Concepts
- **Bootstrapping**: Using current estimates to improve future estimates
- **Online Learning**: Updates happen immediately after each experience
- **Temporal Difference**: Learning from the difference between predictions

## 🔬 Experiments

### Hyperparameter Testing
```python
# Test different learning rates
for alpha in [0.01, 0.1, 0.3, 0.5]:
    agent = TDLearningAgent(num_states=5, alpha=alpha, gamma=0.9)
    agent.train(env, num_episodes=100)
```

### Environment Variations
```python
# Test different environment sizes
for num_states in [3, 5, 10, 20]:
    env = TDLearningEnvironment(num_states=num_states)
    agent = TDLearningAgent(num_states=num_states)
    agent.train(env, num_episodes=200)
```

## 📚 Educational Use

Perfect for:
- **RL Course Assignments** - Clear, well-documented implementation
- **Research Baseline** - Solid foundation for TD learning experiments  
- **Concept Demonstration** - Visual learning of value function convergence
- **Algorithm Comparison** - Benchmark against other RL methods

## 🐛 Troubleshooting

### Common Issues
- **Values not converging**: Check learning rate (try α=0.1)
- **Oscillating values**: Learning rate too high (reduce α)
- **Slow learning**: Learning rate too low (increase α) or more episodes needed
- **Import errors**: Install required packages with pip

### Performance Tips
- **Faster convergence**: Increase learning rate (α) but watch for instability
- **Better exploration**: Implement ε-greedy action selection
- **Larger environments**: Increase episode count proportionally

## 📖 References

- Sutton, R. S., & Barto, A. G. (2018). *Reinforcement Learning: An Introduction* (2nd ed.)
- Chapter 6: Temporal Difference Learning
- [Online Book](http://incompleteideas.net/book/the-book-2nd.html)

## 🤝 Contributing

1. Fork the repository
2. Create feature branch (`git checkout -b feature/amazing-feature`)
3. Commit changes (`git commit -m 'Add amazing feature'`)
4. Push to branch (`git push origin feature/amazing-feature`)
5. Open Pull Request

## 📝 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- Richard Sutton and Andrew Barto for foundational RL theory
- OpenAI Gym for environment design inspiration
- Matplotlib community for visualization tools

---

## 📝 **Blog Post Draft**

# Understanding Temporal Difference Learning: Learning to Predict the Future

*How AI agents learn to estimate value using incomplete information*

## The Problem: Learning Without Complete Information

Imagine you're exploring a new city and trying to figure out which neighborhoods are "good" to be in. Traditional approaches might require you to complete entire walking tours before updating your opinions. But what if you could learn immediately from each step?

That's exactly what Temporal Difference (TD) Learning does for AI agents.

## What Makes TD Learning Special?

Unlike Monte Carlo methods that wait for complete episodes, TD learning updates its beliefs **immediately** after each experience. It's like updating your restaurant ratings after each meal, rather than waiting until you've tried every dish.

### The Magic Formula

V(s) ← V(s) + α[r + γV(s') - V(s)]

This simple equation captures profound learning:
- **V(s)**: "How good do I think this state is?"
- **r + γV(s')**: "What did I just learn about this state?"
- **α**: "How much should I trust this new information?"

## Seeing TD Learning in Action

I implemented a complete TD learning system and watched it learn. Here's what happened:

### Episode 1: First Discoveries
```
Initial values: [0.0, 0.0, 0.0, 0.0, 0.0]
After episode:  [-0.09, 0.0, -0.09, 1.0, 0.0]
```

The agent discovered that state 3 leads to a +10 reward and immediately updated its value!

### Episode 20: Information Spreads
```
Values: [1.57, 4.27, 6.11, 8.88, 0.0]
```

Like ripples in a pond, the value information propagated backwards. States closer to the reward became more valuable.

### Episode 100: Convergence
```
Final values: [2.42, 4.85, 6.91, 8.67, 0.0]
```

Perfect! The agent learned that each state's value reflects its distance from the goal.

## Why This Matters

TD learning is everywhere in modern AI:
- **Game AI**: Learning chess positions without playing complete games
- **Recommendation Systems**: Updating preferences from immediate feedback
- **Autonomous Vehicles**: Learning road conditions from each sensor reading
- **Financial Trading**: Adjusting strategies from each market tick

## Key Insights from Implementation

### 1. Bootstrap Learning Works
The agent successfully learned by using its own imperfect estimates. Like a student who gets better by checking their work against their current best understanding.

### 2. Gradual Convergence
TD errors started large (2.0+) and gradually decreased (1.4-), showing the algorithm naturally converging to correct values.

### 3. Online Learning is Powerful
No waiting for complete episodes meant faster adaptation and more efficient learning.

## The Bigger Picture

TD learning represents a fundamental shift in how we think about learning:
- **From batch to online**: Learn from each experience immediately
- **From certainty to estimation**: Use best current guesses to improve
- **From complete to incremental**: Make progress with partial information

This mirrors how humans actually learn - we don't wait for complete life experiences before updating our beliefs about the world.

## Try It Yourself

The complete implementation is available on GitHub with detailed logging so you can watch every step of the learning process. It's fascinating to see an algorithm bootstrap itself to knowledge!

```python
# Watch TD learning in action
agent = TDLearningAgent(alpha=0.1, gamma=0.9)
agent.train(env, num_episodes=100)
agent.visualize_training()
```

## What's Next?

This simple TD implementation opens doors to:
- **Q-Learning**: Learning optimal actions, not just state values
- **Deep TD Networks**: Using neural networks for complex state spaces
- **Actor-Critic Methods**: Combining TD learning with policy optimization

TD learning isn't just an algorithm - it's a philosophy of learning from incomplete information, which might be the most human thing about artificial intelligence.

---

*Want to dive deeper? Check out the full implementation with step-by-step explanations and visualizations.*

---

## ⚙️ **Requirements File**

```txt
# requirements.txt

# Core scientific computing
numpy>=1.21.0
matplotlib>=3.5.0

# Data handling and analysis
pandas>=1.3.0

# Enhanced visualization (optional)
seaborn>=0.11.0
plotly>=5.0.0

# Jupyter notebook support (optional)
jupyter>=1.0.0
ipywidgets>=7.6.0

# Development tools (optional)
pytest>=6.0.0
black>=21.0.0
flake8>=3.9.0

# Documentation (optional)
sphinx>=4.0.0
sphinx-rtd-theme>=0.5.0
```

---

## 📋 **Installation Instructions**

```bash
# Basic installation
pip install -r requirements.txt

# Or minimal installation
pip install numpy matplotlib

# For development
pip install -r requirements.txt
pip install -e .

# For Google Colab
!pip install numpy matplotlib seaborn pandas plotly
```

---

## 🎯 **Usage Examples**

```python
# examples.py

from td_learning import TDLearningEnvironment, TDLearningAgent
import numpy as np
import matplotlib.pyplot as plt

# Example 1: Basic TD Learning
def basic_example():
    env = TDLearningEnvironment(num_states=5)
    agent = TDLearningAgent(num_states=5, alpha=0.1, gamma=0.9)
    agent.train(env, num_episodes=100)
    agent.visualize_training()
    return agent

# Example 2: Parameter Comparison
def compare_learning_rates():
    results = {}
    learning_rates = [0.01, 0.1, 0.3, 0.5]
    
    for alpha in learning_rates:
        env = TDLearningEnvironment(num_states=5)
        agent = TDLearningAgent(num_states=5, alpha=alpha, gamma=0.9)
        agent.train(env, num_episodes=100)
        results[alpha] = agent.V.copy()
    
    # Plot comparison
    plt.figure(figsize=(10, 6))
    for alpha, values in results.items():
        plt.plot(values, label=f'α={alpha}', marker='o')
    plt.xlabel('State')
    plt.ylabel('Final Value')
    plt.title('Effect of Learning Rate on Final Values')
    plt.legend()
    plt.grid(True)
    plt.show()

# Example 3: Environment Size Study
def environment_size_study():
    sizes = [3, 5, 10, 15]
    convergence_episodes = []
    
    for size in sizes:
        env = TDLearningEnvironment(num_states=size)
        agent = TDLearningAgent(num_states=size, alpha=0.1, gamma=0.9)
        agent.train(env, num_episodes=200)
        
        # Find convergence point (when TD error < 0.1)
        td_errors = agent.training_metrics['avg_td_error']
        convergence = next((i for i, error in enumerate(td_errors) if error < 0.1), 200)
        convergence_episodes.append(convergence)
    
    plt.figure(figsize=(8, 6))
    plt.plot(sizes, convergence_episodes, 'bo-')
    plt.xlabel('Environment Size (Number of States)')
    plt.ylabel('Episodes to Convergence')
    plt.title('Convergence Speed vs Environment Complexity')
    plt.grid(True)
    plt.show()

if __name__ == "__main__":
    # Run examples
    agent = basic_example()
    compare_learning_rates()
    environment_size_study()# TemporalDifferenceLearning
# TemporalDifferenceLearning