README.md

---
license: apache-2.0
language:
- en
base_model:
- microsoft/resnet-50
pipeline_tag: image-classification
---

# Model Card for ChartTypeNet-DocFigure

ChartTypeNet-DocFigure is a deep learning model for scientific chart type classification. Given a single chart image, it predicts the chart type among 28 scientific figure categories. The model is trained on the DocFigure dataset, the largest and most diverse public dataset for document figure classification.

## Model Details

### Model Description

ChartTypeNet-DocFigure automates the classification of scientific chart types in document images. It leverages a ResNet-50 backbone and is trained on 33,000 annotated figures from the DocFigure dataset, covering 28 distinct chart types. This model is intended for use in document image understanding, chart parsing, dataset analysis, and scientific document mining.

- **Developed by:** Hansheng Zhu
- **Model type:** Image Classification
- **License:** Apache-2.0
- **Finetuned from model:** microsoft/resnet-50

### Model Sources

- **Repository:** [https://github.com/hanszhu/ChartSense](https://github.com/hanszhu/ChartSense)
- **Paper:** https://arxiv.org/abs/2106.01841

## Uses

### Direct Use

- Scientific chart type classification from images
- Preprocessing for document image understanding pipelines
- Dataset analysis and mining in scientific literature

### Downstream Use

- As a preprocessing step for chart element detection or information extraction
- Integration into document parsing or digital library systems

### Out-of-Scope Use

- Classification of non-scientific images
- Use on charts outside the 28 DocFigure categories
- Medical or legal decision making

## Bias, Risks, and Limitations

- The model is limited to the 28 chart types present in the DocFigure dataset.
- May not generalize to charts with highly unusual styles or poor image quality.
- Potential dataset bias: DocFigure is sourced from computer vision conference papers.

### Recommendations

Users should verify predictions on out-of-domain data and be aware of the model’s limitations regarding chart style and domain.

## How to Get Started with the Model

```python
import torch
from PIL import Image
from torchvision import transforms

# Load model (example, adjust for your codebase)
model = torch.load('chart_type.pth', map_location='cpu')
model.eval()

# Preprocess image
img = Image.open('example_chart.png').convert('RGB')
transform = transforms.Compose([
    transforms.Resize((224, 224)),  # or as used in your training
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])
input_tensor = transform(img).unsqueeze(0)

# Inference
with torch.no_grad():
    logits = model(input_tensor)
    pred = logits.argmax(dim=1).item()

# Map to label
chart_type_labels = [
    'Line graph', 'Natural image', 'Table', '3D object', 'Bar plot', 'Scatter plot',
    'Medical image', 'Sketch', 'Geographic map', 'Flow chart', 'Heat map', 'Mask',
    'Block diagram', 'Venn diagram', 'Confusion matrix', 'Histogram', 'Box plot',
    'Vector plot', 'Pie chart', 'Surface plot', 'Algorithm', 'Contour plot',
    'Tree diagram', 'Bubble chart', 'Polar plot', 'Area chart', 'Pareto chart', 'Radar chart'
]
print("Predicted chart type:", chart_type_labels[pred])
```

## Training Details

### Training Data

- **Dataset:** [DocFigure](https://arxiv.org/abs/2106.01841)
- 33,000 annotated figures from scientific articles (CVPR, ECCV, ICCV, etc.)
- 28 chart types, manually labeled using a custom web-based tool

### Training Procedure

- Images resized to 224x224
- Normalized using ImageNet statistics
- **Training regime:** fp32
- **Optimizer:** Adam
- **Batch size:** 64
- **Epochs:** 30
- **Learning rate:** 1e-4

## Evaluation

### Testing Data, Factors & Metrics

- **Testing Data:** Held-out split from DocFigure dataset
- **Factors:** Chart type, image quality
- **Metrics:** Accuracy, Similarity-aware Accuracy (SimAcc), Confusion Analysis, F1 score, Precision, Recall

### Results

| Metric           | Train   | Validation |
|------------------|---------|------------|
| Accuracy         | 95.61%  | 86.71%     |
| SimAcc           | 98.70%  | 96.20%     |
| Best Val Acc     |         | 87.03%     |
| Total Loss       | 1.3843  | 2.2681     |
| Base Loss        | 0.887   | 1.073      |
| Sim Loss         | 0.372   | 1.192      |
| Cont Loss        | 0.125   | 0.004      |
| Learning Rate    | 0.000000| 0.000000   |

#### Confusion Analysis
- High-penalty confusion: 16.57%
- S↔B: 8 | L↔A: 30 | B↔H: 93

#### Summary

Enhanced training completed! The model achieves a best validation accuracy of **87.03%** and a final similarity-aware accuracy (SimAcc) of **96.20%**. Training accuracy is **95.61%** (SimAcc: **98.70%**). Confusion analysis shows high-penalty confusion at 16.57%, with most confusion between similar chart types (S↔B, L↔A, B↔H). The model demonstrates strong generalization and robustness on the DocFigure validation set.

## Environmental Impact

- **Hardware Type:** NVIDIA V100 GPU
- **Hours used:** 8
- **Cloud Provider:** Google Cloud
- **Compute Region:** us-central1
- **Carbon Emitted:** ~12 kg CO2eq (estimated)

## Technical Specifications

### Model Architecture and Objective

- ResNet-50 backbone
- Image classification head for 28 classes

### Compute Infrastructure

- **Hardware:** NVIDIA V100 GPU
- **Software:** PyTorch 1.13, torchvision 0.14, Python 3.9

## Citation

**BibTeX:**

```bibtex
@article{DocFigure2021,
  title={DocFigure: A Dataset for Scientific Figure Classification},
  author={S. Afzal, et al.},
  journal={arXiv preprint arXiv:2106.01841},
  year={2021}
}
```

**APA:**

Afzal, S., et al. (2021). DocFigure: A Dataset for Scientific Figure Classification. arXiv preprint arXiv:2106.01841.

## Glossary

- **Chart Type:** The category of a scientific figure (e.g., bar plot, scatter plot, etc.)

## More Information

- [DocFigure Paper](https://arxiv.org/abs/2106.01841)

## Model Card Authors

Hansheng Zhu

## Model Card Contact

[email protected]