|
--- |
|
language: en |
|
tags: |
|
- topic-drift |
|
- conversation-analysis |
|
- pytorch |
|
- attention |
|
license: mit |
|
datasets: |
|
- leonvanbokhorst/topic-drift-v2 |
|
metrics: |
|
- rmse |
|
- r2_score |
|
model-index: |
|
- name: topic-drift-detector |
|
results: |
|
- task: |
|
type: topic-drift-detection |
|
name: Topic Drift Detection |
|
dataset: |
|
name: leonvanbokhorst/topic-drift-v2 |
|
type: conversations |
|
metrics: |
|
- name: Test RMSE |
|
type: rmse |
|
value: 0.0144 |
|
- name: Test R² |
|
type: r2 |
|
value: 0.8666 |
|
- name: Test Loss |
|
type: loss |
|
value: 0.0002 |
|
--- |
|
|
|
# Topic Drift Detector Model |
|
|
|
## Version: v20241226_105737 |
|
|
|
This model detects topic drift in conversations using a streamlined attention-based architecture. Trained on the [leonvanbokhorst/topic-drift-v2](https://huggingface.co/datasets/leonvanbokhorst/topic-drift-v2) dataset. |
|
|
|
## Model Architecture |
|
- Efficient single-layer attention mechanism |
|
- Direct pattern recognition |
|
- Streamlined processing pipeline |
|
- Optimized scaling factor (4.0) |
|
- PreNorm layers with residual connections |
|
|
|
### Key Components: |
|
1. **Embedding Processor**: |
|
- Input dimension: 1024 |
|
- Hidden dimension: 512 |
|
- Dropout rate: 0.35 |
|
- PreNorm layers with residual connections |
|
|
|
2. **Attention Block**: |
|
- Single attention layer |
|
- Feed-forward dimension: 512 |
|
- Learned position encodings |
|
- Residual connections |
|
|
|
3. **Pattern Recognition**: |
|
- Direct feature extraction |
|
- Efficient tensor operations |
|
- Optimized memory usage |
|
|
|
## Performance Metrics |
|
```txt |
|
=== Full Training Results === |
|
Best Validation RMSE: 0.0142 |
|
Best Validation R²: 0.8711 |
|
|
|
=== Test Set Results === |
|
Loss: 0.0002 |
|
RMSE: 0.0144 |
|
R²: 0.8666 |
|
``` |
|
|
|
## Training Details |
|
- Dataset: 6400 conversations (5120 train, 640 val, 640 test) |
|
- Window size: 8 turns |
|
- Batch size: 32 |
|
- Learning rate: 0.0001 |
|
- Early stopping patience: 15 |
|
- Distribution regularization weight: 0.1 |
|
- Target standard deviation: 0.2 |
|
- Base embeddings: BAAI/bge-m3 |
|
|
|
## Key Improvements |
|
1. **Simplified Architecture**: |
|
- Reduced complexity |
|
- Focused pattern detection |
|
- Efficient processing |
|
- Optimized memory usage |
|
|
|
2. **Performance Benefits**: |
|
- Improved RMSE (0.0144) |
|
- Strong R² score (0.8666) |
|
- Consistent predictions |
|
- Wide score range |
|
|
|
## Usage Example |
|
```python |
|
import torch |
|
from transformers import AutoModel, AutoTokenizer |
|
|
|
# Load base embedding model |
|
base_model = AutoModel.from_pretrained('BAAI/bge-m3') |
|
tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-m3') |
|
|
|
# Load topic drift detector |
|
model = torch.load('models/v20241226_105737/topic_drift_model.pt') |
|
model.eval() |
|
|
|
# Prepare conversation window (8 turns) |
|
conversation = [ |
|
"How was your weekend?", |
|
"It was great! Went hiking.", |
|
"Which trail did you take?", |
|
"The mountain loop trail.", |
|
"That's nice. By the way, did you watch the game?", |
|
"Yes! What an amazing match!", |
|
"The final score was incredible.", |
|
"I couldn't believe that last-minute goal." |
|
] |
|
|
|
# Get embeddings |
|
with torch.no_grad(): |
|
inputs = tokenizer(conversation, padding=True, truncation=True, return_tensors='pt') |
|
embeddings = base_model(**inputs).last_hidden_state.mean(dim=1) # [8, 1024] |
|
|
|
# Reshape for model input [1, 8*1024] |
|
conversation_embeddings = embeddings.view(1, -1) |
|
|
|
# Get drift score |
|
drift_scores = model(conversation_embeddings) |
|
|
|
print(f"Topic drift score: {drift_scores.item():.4f}") |
|
# Higher scores indicate more topic drift |
|
``` |
|
|
|
## Limitations |
|
- Works best with English conversations |
|
- Requires exactly 8 turns of conversation |
|
- Each turn should be between 1-512 tokens |
|
- Relies on BAAI/bge-m3 embeddings |
|
|
|
## Training Curves |
|
 |
|
|
