Update README.md

README.md

@@ -4,12 +4,33 @@ license: apache-2.0
 base_model: answerdotai/ModernBERT-base
 tags:
 - generated_from_trainer
+- text-classification
+- topic-detection
+- modernbert
+- web-content-classification
 metrics:
 - accuracy
 - f1
+- worst_group_accuracy
 model-index:
-- name: modernbert-topics-1m
-  results: []
+- name: davanstrien/modernbert-topics-1m
+  results:
+  - task:
+      type: text-classification
+      name: Topic Classification
+    dataset:
+      name: WebOrganizer/TopicAnnotations-Llama-3.1-8B
+      type: WebOrganizer/TopicAnnotations-Llama-3.1-8B
+    metrics:
+      - name: Accuracy
+        type: accuracy
+        value: 0.7949
+      - name: F1
+        type: f1
+        value: 0.7948
+      - name: Worst Group Accuracy
+        type: worst_group_accuracy
+        value: 0.5723
 datasets:
 - WebOrganizer/TopicAnnotations-Llama-3.1-8B
 language:
@@ -17,68 +38,157 @@ language:
 pipeline_tag: text-classification
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
-# modernbert-topics-1m
-
-This model is a fine-tuned version of [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) on the [WebOrganizer/TopicAnnotations-Llama-3.1-8B](https://huggingface.co/datasets/WebOrganizer/TopicAnnotations-Llama-3.1-8B) dataset. 
-
-It achieves the following results on the evaluation set:
-- Loss: 0.5923
-- Accuracy: 0.7949
-- F1: 0.7948
-- Worst Group Accuracy: 0.5723
-
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
-
-### Training hyperparameters
-
-The following hyperparameters were used during training:
-- learning_rate: 5e-05
-- train_batch_size: 64
-- eval_batch_size: 64
-- seed: 42
-- distributed_type: multi-GPU
-- num_devices: 4
-- gradient_accumulation_steps: 4
-- total_train_batch_size: 1024
-- total_eval_batch_size: 256
-- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: linear
-- lr_scheduler_warmup_steps: 5000
-- num_epochs: 5
-
-### Training results
-
-| Training Loss | Epoch  | Step | Validation Loss | Accuracy | F1     | Worst Group Accuracy |
-|:-------------:|:------:|:----:|:---------------:|:--------:|:------:|:--------------------:|
-| 6.7028        | 0.5223 | 500  | 0.8360          | 0.7279   | 0.7269 | 0.4539               |
-| 2.9939        | 1.0439 | 1000 | 0.7108          | 0.7635   | 0.7632 | 0.5876               |
-| 2.625         | 1.5662 | 1500 | 0.6393          | 0.7785   | 0.7778 | 0.6283               |
-| 2.425         | 2.0878 | 2000 | 0.6043          | 0.7886   | 0.7880 | 0.6098               |
-| 2.2422        | 2.6101 | 2500 | 0.5870          | 0.7908   | 0.7902 | 0.6272               |
-| 2.158         | 3.1316 | 3000 | 0.5723          | 0.7944   | 0.7939 | 0.6427               |
-| 1.9898        | 3.6540 | 3500 | 0.5684          | 0.7947   | 0.7945 | 0.6595               |
-| 1.8709        | 4.1755 | 4000 | 0.6023          | 0.7941   | 0.7938 | 0.6061               |
-| 1.6459        | 4.6978 | 4500 | 0.5923          | 0.7949   | 0.7948 | 0.5723               |
-
-
-### Framework versions
-
-- Transformers 4.51.3
-- Pytorch 2.6.0+cu124
-- Datasets 3.5.0
-- Tokenizers 0.21.1
+# ModernBERT Topics Classification Model (davanstrien/modernbert-topics-1m)
+
+## Model Description
+
+This model is a fine-tuned version of [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) on the [WebOrganizer/TopicAnnotations-Llama-3.1-8B](https://huggingface.co/datasets/WebOrganizer/TopicAnnotations-Llama-3.1-8B) dataset for multi-class topic classification. It is designed to classify web content into 24 distinct topic categories, ranging from "Adult Content" to "Food & Dining," making it useful for content categorization, filtering, and organization tasks.
+
+The model leverages ModernBERT's architecture, which includes efficient attention mechanisms and allows for longer context handling compared to traditional BERT models (up to 8192 tokens). This implementation was specifically created to be compatible with VLLM, enabling faster and more efficient deployment, especially for processing large volumes of web content.
+
+This model serves as an alternative to the original [WebOrganizer/TopicClassifier](https://huggingface.co/WebOrganizer/TopicClassifier), with the key difference being the use of ModernBERT as the base architecture instead of typical BERT models, providing improved efficiency and longer context handling.
+
+## Performance
+
+The model achieves the following results on the evaluation set:
+- **Loss:** 0.5923
+- **Accuracy:** 0.7949
+- **F1 Score:** 0.7948
+- **Worst Group Accuracy:** 0.5723
+
+These metrics indicate strong overall performance, with nearly 80% accuracy across all categories. The "Worst Group Accuracy" metric of 57.23% suggests there is some variance in performance across different topic categories, which should be considered when using this model for specific domains.
+
+## Intended Uses & Limitations
+
+### Intended Uses
+- Web content categorization and organization
+- Content filtering systems for various platforms and applications
+- Topic-based content recommendation systems
+- Research and analysis of web content distribution
+- Automated content tagging for content management systems
+- Information retrieval systems that benefit from topical categorization
+- Pre-processing step for domain-specific training data curation
+
+### Limitations
+- Performance varies across categories, with a worst group accuracy of 57.23%, indicating some topics may be classified less reliably than others
+- The model may struggle with content that spans multiple categories or contains ambiguous topics
+- Limited to English language content
+- May not perform optimally on specialized domain-specific content that differs significantly from the training data
+- Classification is limited to the 24 predefined categories; content outside these categories may be misclassified
+- The model's training data was annotated by an LLM (Llama-3.1-8B), which may introduce systematic biases compared to human annotations
+- While the model can process up to 8192 tokens, very long documents may lose important context if truncated
+
+## Training and Evaluation Data
+
+This model was trained on the [WebOrganizer/TopicAnnotations-Llama-3.1-8B](https://huggingface.co/datasets/WebOrganizer/TopicAnnotations-Llama-3.1-8B) dataset, which contains 1 million web pages annotated with topic labels generated by the Llama-3.1-8B model. The dataset is derived from the DCLM RefinedWeb reproduction and was created as part of the research presented in ["Organize the Web: Constructing Domains Enhances Pre-Training Data Curation"](https://arxiv.org/abs/2502.10341).
+
+Each sample in the dataset contains the full text content of a web page, its URL, the most likely topic label with its probability, probabilities for all possible topics, and additional metadata. The dataset was specifically designed for training topic classifiers and is used as first-stage training data for the WebOrganizer TopicClassifier.
+
+The 24 topic categories covered by this model are:
+
+1. Adult Content
+2. Politics (includes social issues, campaigns, legislation, geopolitics, protests, activism)
+3. History & Geography (includes archaeology)
+4. Health (includes medicine, wellness, mental health, veterinary science, nutrition)
+5. Home & Hobbies (includes real estate, DIY, gardening, pets, collecting)
+6. Travel & Tourism (includes hospitality, hotels, cruises)
+7. Religion (includes spirituality)
+8. Sports & Fitness (includes martial arts, motor sports, outdoor activities)
+9. Games (includes video games, board games, gambling)
+10. Entertainment (includes music, movies, TV, celebrities, humor)
+11. Literature (includes criticism, linguistics, philosophy, humanities)
+12. Art & Design (includes architecture)
+13. Science, Math & Technology (includes physics, chemistry, biology, mathematics, engineering)
+14. Education & Jobs (includes pedagogy, training, academia)
+15. Software Development (includes algorithms, coding, web development)
+16. Fashion & Beauty (includes clothing, accessories, cosmetics)
+17. Industrial (includes mining, agriculture, manufacturing, construction)
+18. Software (topics related to software use and the internet)
+19. Finance & Business (includes taxes, investments, insurance, marketing, HR)
+20. Electronics & Hardware (includes computer hardware, phones, consumer electronics)
+21. Crime & Law (includes law enforcement)
+22. Transportation (includes vehicles, public transit, aviation, logistics)
+23. Social Life (includes family, relationships, community)
+24. Food & Dining (includes recipes, groceries, beverages, restaurants)
+
+## Training Procedure
+
+### Training Hyperparameters
+
+The model was trained with the following hyperparameters:
+- **Learning rate:** 5e-05
+- **Train batch size:** 64 (1024 total with gradient accumulation)
+- **Eval batch size:** 64 (256 total)
+- **Optimizer:** AdamW with betas=(0.9, 0.999) and epsilon=1e-08
+- **LR scheduler:** Linear with 5000 warmup steps
+- **Training epochs:** 5
+- **Distributed training:** Multi-GPU with 4 devices
+- **Gradient accumulation steps:** 4
+- **Seed:** 42
+
+
+## Technical Specifications
+
+### Model Architecture
+- **Base model:** ModernBertForSequenceClassification
+- **Hidden size:** 768
+- **Number of hidden layers:** 22
+- **Number of attention heads:** 12
+- **Intermediate size:** 1152
+- **Max position embeddings:** 8192
+- **Vocabulary size:** 50368
+
+### Framework Versions
+- Transformers: 4.51.3
+- PyTorch: 2.6.0+cu124
+- Datasets: 3.5.0
+- Tokenizers: 0.21.1
+
+## Inference Information
+
+This model is compatible with VLLM and inference engines, which can significantly improve inference speed, especially for batch processing. When using the model, be sure to use the ModernBERT tokenizer and respect the model's maximum sequence length of 8192 tokens.
+
+Example usage:
+```python
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+import torch
+
+# Load model and tokenizer
+model_name = "davanstrien/modernbert-topics-1m"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForSequenceClassification.from_pretrained(model_name)
+
+# Prepare input text
+text = "The impact of global warming on coral reef ecosystems"
+
+# Tokenize and predict
+inputs = tokenizer(text, return_tensors="pt", truncation=True)
+with torch.no_grad():
+    outputs = model(**inputs)
+    
+# Get prediction
+prediction = outputs.logits.argmax(-1).item()
+predicted_label = model.config.id2label[prediction]
+print(f"Predicted topic: {predicted_label}")
+```
+
+## Ethical Considerations and Biases
+
+- This model may inherit biases present in the training data, potentially leading to inconsistent classification across different demographic or cultural contexts.
+- Topics with less representation in the training data may show lower accuracy.
+- Users should be aware that fully automated content classification without human oversight may lead to inappropriate categorizations in edge cases.
+
+## Citation and Contact Information
+
+If you use this model in your research or applications, please cite the original ModernBERT model, as well as the WebOrganizer dataset and paper:
+
+```bibtex
+@article{wettig2025organize,
+  title={Organize the Web: Constructing Domains Enhances Pre-Training Data Curation},
+  author={Alexander Wettig and Kyle Lo and Sewon Min and Hannaneh Hajishirzi and Danqi Chen and Luca Soldaini},
+  journal={arXiv preprint arXiv:2502.10341},
+  year={2025}
+}
+```
+
+For questions, issues, or contributions related to this model, please reach out through the [Hugging Face model repository](https://huggingface.co/davanstrien/modernbert-topics-1m).