Upload MammoEnsemble

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

config.json

@@ -0,0 +1,37 @@
+{
+  "architectures": [
+    "MammoEnsemble"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration.MammoConfig",
+    "AutoModel": "modeling.MammoEnsemble"
+  },
+  "backbone": "tf_efficientnetv2_s",
+  "dropout": 0.1,
+  "feature_dim": 1280,
+  "image_sizes": [
+    [
+      2048,
+      1024
+    ],
+    [
+      1920,
+      1280
+    ],
+    [
+      1536,
+      1536
+    ]
+  ],
+  "in_chans": 1,
+  "model_type": "mammo",
+  "num_classes": 5,
+  "num_models": 3,
+  "pad_to_aspect_ratio": [
+    true,
+    true,
+    false
+  ],
+  "torch_dtype": "float32",
+  "transformers_version": "4.47.0"
+}

configuration.py

@@ -0,0 +1,32 @@
+from transformers import PretrainedConfig
+from typing import List, Tuple
+
+
+class MammoConfig(PretrainedConfig):
+    model_type = "mammo"
+
+    def __init__(
+        self,
+        backbone: str = "tf_efficientnetv2_s",
+        feature_dim: int = 1280,
+        dropout: float = 0.1,
+        num_classes: int = 5,
+        in_chans: int = 1,
+        num_models: int = 3,
+        image_sizes: List[Tuple[int, int]] = [(2048, 1024), (1920, 1280), (1536, 1536)],
+        pad_to_aspect_ratio: List[bool] = [True, True, False],
+        **kwargs,
+    ):
+        self.backbone = backbone
+        self.feature_dim = feature_dim
+        self.dropout = dropout
+        self.num_classes = num_classes
+        self.in_chans = in_chans
+        self.num_models = num_models
+        assert len(image_sizes) == len(pad_to_aspect_ratio) == num_models, (
+            f"length of `image_sizes` [{len(image_sizes)}] and `pad_to_aspect_ratio` "
+            f"[{len(pad_to_aspect_ratio)}] must be equal to `num_models` [{num_models}]."
+        )
+        self.image_sizes = image_sizes
+        self.pad_to_aspect_ratio = pad_to_aspect_ratio
+        super().__init__(**kwargs)

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ace0998e2d534b08eaa673fad37df16eaffef6dca3f3b5ac2196857a2949596
+size 244305924

modeling.py

@@ -0,0 +1,195 @@
+import albumentations as A
+import numpy as np
+import torch
+import torch.nn as nn
+
+from transformers import PreTrainedModel
+from timm import create_model
+from typing import Mapping, Sequence, Tuple
+from .configuration import MammoConfig
+
+
+def _pad_to_aspect_ratio(img: np.ndarray, aspect_ratio: float) -> np.ndarray:
+    """
+    Pads to specified aspect ratio, only if current aspect ratio is
+    greater.
+    """
+    h, w = img.shape[:2]
+    if h / w > aspect_ratio:
+        new_w = round(h / aspect_ratio)
+        w_diff = new_w - w
+        left_pad = w_diff // 2
+        right_pad = w_diff - left_pad
+        padding = ((0, 0), (left_pad, right_pad))
+        if img.ndim == 3:
+            padding = padding + ((0, 0),)
+        img = np.pad(img, padding, mode="constant", constant_values=0)
+    return img
+
+
+def _to_torch_tensor(x: np.ndarray, device: str) -> torch.Tensor:
+    if x.ndim == 2:
+        x = torch.from_numpy(x).unsqueeze(0)
+    elif x.ndim == 3:
+        x = torch.from_numpy(x)
+        if torch.tensor(x.size()).argmin().item() == 2:
+            # channels last -> first
+            x = x.permute(2, 0, 1)
+    else:
+        raise ValueError(f"Expected 2 or 3 dimensions, got {x.ndim}")
+    return x.float().to(device)
+
+
+class MammoModel(nn.Module):
+    def __init__(
+        self,
+        backbone: str,
+        image_size: Tuple[int, int],
+        pad_to_aspect_ratio: bool,
+        feature_dim: int = 1280,
+        dropout: float = 0.1,
+        num_classes: int = 5,
+        in_chans: int = 1,
+    ):
+        super().__init__()
+        self.backbone = create_model(
+            model_name=backbone,
+            pretrained=False,
+            num_classes=0,
+            global_pool="",
+            features_only=False,
+            in_chans=in_chans,
+        )
+        self.pooling = nn.AdaptiveAvgPool2d(1)
+        self.dropout = nn.Dropout(p=dropout)
+        self.linear = nn.Linear(feature_dim, num_classes)
+
+        self.pad_to_aspect_ratio = pad_to_aspect_ratio
+        self.aspect_ratio = image_size[0] / image_size[1]
+        if self.pad_to_aspect_ratio:
+            self.resize = A.Resize(image_size[0], image_size[1], p=1)
+        else:
+            self.resize = A.Compose(
+                [
+                    A.LongestMaxSize(image_size[0], p=1),
+                    A.PadIfNeeded(image_size[0], image_size[1], p=1),
+                ],
+                p=1,
+            )
+
+    def normalize(self, x: torch.Tensor) -> torch.Tensor:
+        # [0, 255] -> [-1, 1]
+        mini, maxi = 0.0, 255.0
+        x = (x - mini) / (maxi - mini)
+        x = (x - 0.5) * 2.0
+        return x
+
+    def preprocess(
+        self,
+        x: Mapping[str, np.ndarray] | Sequence[Mapping[str, np.ndarray]],
+        device: str,
+    ) -> Sequence[Mapping[str, torch.Tensor]]:
+        # x is a dict (or list of dicts) with keys "cc" and/or "mlo"
+        # though the actual keys do not matter
+        if not isinstance(x, Sequence):
+            assert isinstance(x, Mapping)
+            x = [x]
+        if self.pad_to_aspect_ratio:
+            x = [
+                {
+                    k: _pad_to_aspect_ratio(v.copy(), self.aspect_ratio)
+                    for k, v in sample.items()
+                }
+                for sample in x
+            ]
+        x = [
+            {
+                k: _to_torch_tensor(self.resize(image=v)["image"], device=device)
+                for k, v in sample.items()
+            }
+            for sample in x
+        ]
+        return x
+
+    def forward(
+        self, x: Sequence[Mapping[str, torch.Tensor]]
+    ) -> Mapping[str, torch.Tensor]:
+        batch_tensor = []
+        batch_indices = []
+        for idx, sample in enumerate(x):
+            for k, v in sample.items():
+                batch_tensor.append(v)
+                batch_indices.append(idx)
+
+        batch_tensor = torch.stack(batch_tensor, dim=0)
+        batch_tensor = self.normalize(batch_tensor)
+        features = self.pooling(self.backbone(batch_tensor))
+        b, d = features.shape[:2]
+        features = features.reshape(b, d)
+        logits = self.linear(features)
+        # cancer
+        logits0 = logits[:, 0].sigmoid()
+        # density
+        logits1 = logits[:, 1:].softmax(dim=1)
+        # mean over views
+        batch_indices = torch.tensor(batch_indices)
+        logits0 = torch.stack(
+            [logits0[batch_indices == i].mean(dim=0) for i in batch_indices.unique()]
+        )
+        logits1 = torch.stack(
+            [logits1[batch_indices == i].mean(dim=0) for i in batch_indices.unique()]
+        )
+        return {"cancer": logits0, "density": logits1}
+
+
+class MammoEnsemble(PreTrainedModel):
+    config_class = MammoConfig
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.num_models = config.num_models
+        for i in range(self.num_models):
+            setattr(
+                self,
+                f"net{i}",
+                MammoModel(
+                    config.backbone,
+                    config.image_sizes[i],
+                    config.pad_to_aspect_ratio[i],
+                    config.feature_dim,
+                    config.dropout,
+                    config.num_classes,
+                    config.in_chans,
+                ),
+            )
+
+    @staticmethod
+    def load_image_from_dicom(path: str) -> np.ndarray | None:
+        try:
+            from pydicom import dcmread
+            from pydicom.pixels import apply_voi_lut
+        except ModuleNotFoundError:
+            print("`pydicom` is not installed, returning None ...")
+            return None
+        dicom = dcmread(path)
+        arr = apply_voi_lut(dicom.pixel_array, dicom)
+        if dicom.PhotometricInterpretation == "MONOCHROME1":
+            arr = arr.max() - arr
+
+        arr = arr - arr.min()
+        arr = arr / arr.max()
+        arr = (arr * 255).astype("uint8")
+        return arr
+
+    def forward(
+        self,
+        x: Mapping[str, np.ndarray] | Sequence[Mapping[str, np.ndarray]],
+        device: str = "cpu",
+    ) -> Mapping[str, torch.Tensor]:
+        out = []
+        for i in range(self.num_models):
+            model = getattr(self, f"net{i}")
+            x_pp = model.preprocess(x, device=device)
+            out.append(model(x_pp))
+        out = {k: torch.stack([o[k] for o in out]).mean(0) for k in out[0].keys()}
+        return out