|
|
--- |
|
|
license: apache-2.0 |
|
|
pipeline_tag: image-feature-extraction |
|
|
library_name: transformers |
|
|
tags: |
|
|
- earth-observation |
|
|
- remote-sensing |
|
|
- foundation-model |
|
|
- multi-sensor |
|
|
--- |
|
|
|
|
|
<p align="center"> |
|
|
<img src="imgs/logo.png" alt="Spectral Coverage" width="400"/> |
|
|
</p> |
|
|
|
|
|
# Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images |
|
|
[](https://arxiv.org/abs/2506.19585) |
|
|
[](https://huggingface.co/collections/gsumbul/smarties-685888bb5ecded3f802cc945) |
|
|
[](https://opensource.org/licenses/Apache-2.0) |
|
|
 |
|
|
 |
|
|
[](https://gsumbul.github.io/SMARTIES/) |
|
|
|
|
|
## 🚀 Introduction |
|
|
|
|
|
<!-- Spectral coverage figure --> |
|
|
<p align="center"> |
|
|
<img src="imgs/spectra_fig.png" alt="Spectral Coverage" width="600"/> |
|
|
</p> |
|
|
|
|
|
From optical sensors to microwave radars, leveraging the complementary strengths of remote sensing (RS) sensors is crucial for achieving dense spatio-temporal monitoring of our planet, but recent foundation models (FMs) are often specific to single sensors or to fixed combinations. |
|
|
|
|
|
SMARTIES is a generic and versatile FM lifting sensor-dependent efforts and enabling scalability and generalization to diverse RS sensors: SMARTIES projects data from heterogeneous sensors into a shared spectrum-aware space, enabling the use of arbitrary combinations of bands both for training and inference. To obtain sensor-agnostic representations, SMARTIES was trained as a single, unified transformer model reconstructing masked multi-sensor data with cross-sensor token mixup, while modulating its feature representations to accept diverse sensors as input. |
|
|
|
|
|
## ✨ Key Features |
|
|
- 🛰️ **Multi-Sensor Representations:** SMARTIES enables sensor-agnostic processing of Earth observation data, including optical (e.g., Sentinel-2), radar (e.g., Sentinel-1), and sub-meter resolution RGB (e.g., Maxar) imagery and unseen ones in a zero-shot manner. |
|
|
- 🌈 **Spectrum-Aware Projections:** SMARTIES projects data from heterogeneous sensors into a shared spectrum-aware space: given a specific sensor, each one of its bands is projected by projection layers specific to wavelength ranges. |
|
|
- ⚡ **Lightweight and Scalable:** SMARTIES is designed to be lightweight and scalable, making it suitable for a wide range of remote sensing applications. |
|
|
- 🔀 **Flexible Band Combinations:** SMARTIES can handle arbitrary combinations of spectral bands from different sensors, enabling flexible remote sensing applications. |
|
|
- 🔄 **Downstream Transfer:** SMARTIES enables downstream transfer using a unified model across a diverse set of sensors and tasks, including scene classification, semantic segmentation, and multi-label classification. |
|
|
|
|
|
<!-- Model architecture figure --> |
|
|
<p align="center"> |
|
|
<img src="imgs/model_fig.png" alt="SMARTIES Model Architecture" width="700"/> |
|
|
</p> |
|
|
|
|
|
This repository contains the model weights of SMARTIES (ViT-B). |
|
|
|
|
|
For usage instructions, dataset details, and full documentation, please visit the [SMARTIES GitHub page](https://github.com/gsumbul/SMARTIES). The details of SMARTIES are described in our paper, available on [arXiv](https://arxiv.org/abs/2506.19585). |
|
|
|
|
|
## 📣 Attribution |
|
|
If you use SMARTIES, please cite the paper: |
|
|
|
|
|
``` |
|
|
@article{smarties, |
|
|
title={{SMARTIES}: Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images}, |
|
|
author={Gencer Sumbul and Chang Xu and Emanuele Dalsasso and Devis Tuia}, |
|
|
journal={arXiv preprint arXiv:2506.19585}, |
|
|
year={2025} |
|
|
} |
|
|
``` |
|
|
|
|
|
## 📄 License |
|
|
This repository is released under the Apache v2 License. |
|
|
|
|
|
## 🙏 Acknowledgements |
|
|
SMARTIES is supported by the European Space Agency (ESA) through the Discovery and Preparation Program, and is part of the project Toward a Foundation Model for Multi-Sensor Earth Observation Data with Language Semantics. |
