Add comprehensive model card for OMAR-RQ
Browse filesThis PR adds a comprehensive model card for the OMAR-RQ model, linking it to the paper [OMAR-RQ: Open Music Audio Representation Model Trained with Multi-Feature Masked Token Prediction](https://huggingface.co/papers/2507.03482).
Key additions include:
- A link to the official GitHub repository: https://github.com/MTG/OMAR-RQ
- Relevant metadata: `pipeline_tag: audio-classification`, `library_name: omar_rq`, `license: cc-by-nc-sa-4.0`, and additional `tags` for better discoverability.
- The paper's abstract and detailed sample usage for inference.
- A table of available OMAR-RQ models with links to their respective Hugging Face Hub pages.
- Clear licensing information and a BibTeX citation.
This will significantly improve the discoverability and usability of the model on the Hugging Face Hub.
|
@@ -0,0 +1,77 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
pipeline_tag: audio-classification
|
| 3 |
+
library_name: omar_rq
|
| 4 |
+
license: cc-by-nc-sa-4.0
|
| 5 |
+
tags:
|
| 6 |
+
- audio-feature-extraction
|
| 7 |
+
- music
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
+
# OMAR-RQ: Open Music Audio Representation Model Trained with Multi-Feature Masked Token Prediction
|
| 11 |
+
|
| 12 |
+
This repository contains the model weights for **OMAR-RQ**, as presented in the paper [OMAR-RQ: Open Music Audio Representation Model Trained with Multi-Feature Masked Token Prediction](https://huggingface.co/papers/2507.03482).
|
| 13 |
+
|
| 14 |
+
## Abstract
|
| 15 |
+
Developing open-source foundation models is essential for advancing research in music audio understanding and ensuring access to powerful, multipurpose representations for music information retrieval. We present OMAR-RQ, a model trained with self-supervision via masked token classification methodologies using a large-scale dataset with over 330,000 hours of music audio. We experiment with different input features and quantization options, and achieve state-of-the-art performance in music tagging, pitch estimation, chord recognition, beat tracking, segmentation, and difficulty estimation among open self-supervised models. We open-source our training and evaluation pipelines and model weights, available at this https URL .
|
| 16 |
+
|
| 17 |
+
## Code
|
| 18 |
+
The training, validation, and inference code, along with further details, is available at the official GitHub repository: [https://github.com/MTG/OMAR-RQ](https://github.com/MTG/OMAR-RQ).
|
| 19 |
+
|
| 20 |
+
## Inference
|
| 21 |
+
You can load an OMAR-RQ model directly by specifying its Hugging Face model ID. First, install the library:
|
| 22 |
+
|
| 23 |
+
```bash
|
| 24 |
+
pip install omar-rq
|
| 25 |
+
```
|
| 26 |
+
|
| 27 |
+
Then, use the following Python code for embedding extraction:
|
| 28 |
+
|
| 29 |
+
```python
|
| 30 |
+
import torch
|
| 31 |
+
from omar_rq import get_model
|
| 32 |
+
|
| 33 |
+
# Embedding extraction example
|
| 34 |
+
x = torch.randn(1, 16000 * 4).cpu() # Example audio input (batch_size, samples)
|
| 35 |
+
|
| 36 |
+
model_id = "mtg-upf/omar-rq-multifeature-25hz-fsq" # This repository's model ID
|
| 37 |
+
model = get_model(model_id=model_id, device="cpu")
|
| 38 |
+
|
| 39 |
+
embeddings = model.extract_embeddings(x, layers=[6])
|
| 40 |
+
|
| 41 |
+
timestamps = torch.arange(embeddings.shape[2]) / model.eps
|
| 42 |
+
|
| 43 |
+
print(f"Extracted embeddings shape: {embeddings.shape}")
|
| 44 |
+
print(f"Number of timestamps: {len(timestamps)}")
|
| 45 |
+
```
|
| 46 |
+
|
| 47 |
+
For more details on `get_model` and `extract_embeddings` usage, please refer to the [GitHub repository](https://github.com/MTG/OMAR-RQ).
|
| 48 |
+
|
| 49 |
+
## Available Models
|
| 50 |
+
OMAR-RQ models are offered in different configurations, each with its own strengths and weaknesses.
|
| 51 |
+
Models based on mel spectrogram (**base** and **multicodebook**) tend to perform better on semantic tasks such as auto-tagging, structure recognition, and difficulty estimation.
|
| 52 |
+
On the other hand, **multifeature-24hz-fsq** offers the best performance in tonal and temporal tasks such as pitch and chord estimation, and beat tracking.
|
| 53 |
+
|
| 54 |
+
| Model | Hugging Face ID | Input | Rate | Tagging | Difficulty | Pitch | Chord | Beat | Structure |
|
| 55 |
+
|---|---|---|---|---|---|---|---|---|---|
|
| 56 |
+
| | | Hz | _mAP_ | _MSE_ | _acc._ | _acc._ | _F1_ | _acc._ |
|
| 57 |
+
| **base** | [mtg-upf/omar-rq-base](https://huggingface.co/mtg-upf/omar-rq-base) | mel | 15.63 | .482 | **1.65** | .892 | .657 | .783 | **.647** |
|
| 58 |
+
| **multicodebook** | [mtg-upf/omar-rq-multicodebook](https://huggingface.co/mtg-upf/omar-rq-multicodebook) | mel | 15.63 | **.488** | 1.66 | .897 | .675 | .775 | .639 |
|
| 59 |
+
| **multifeature** | [mtg-upf/omar-rq-multifeature](https://huggingface.co/mtg-upf/omar-rq-multifeature) | audio | 18.75 | .467 | 1.76 | .938 | .734 | .833 | .623 |
|
| 60 |
+
| **multifeature-25hz** | [mtg-upf/omar-rq-multifeature-25hz](https://huggingface.co/mtg-upf/omar-rq-multifeature-25hz) | audio | 25 | .463 | 1.79 | .932 | .728 | .848 | .628 |
|
| 61 |
+
| **multifeature-25hz-fsq**| [mtg-upf/omar-rq-multifeature-25hz-fsq](https://huggingface.co/mtg-upf/omar-rq-multifeature-25hz-fsq) | audio | 25 | .463 | 1.71 | **.940**| **.749**| **.855** | .628 |
|
| 62 |
+
|
| 63 |
+
## License
|
| 64 |
+
The code in the [OMAR-RQ GitHub repository](https://github.com/MTG/OMAR-RQ) is available under the [AGPL-3.0 license](https://www.gnu.org/licenses/agpl-3.0.en.html).
|
| 65 |
+
The model weights on this Hugging Face Hub are released under the [CC BY-NC-SA 4.0 license](https://creativecommons.org/licenses/by-nc-sa/4.0/) for non-commercial applications.
|
| 66 |
+
|
| 67 |
+
## Citation
|
| 68 |
+
If you find this work helpful or inspiring, please feel free to cite it using the following BibTeX entry:
|
| 69 |
+
|
| 70 |
+
```bibtex
|
| 71 |
+
@article{fust,
|
| 72 |
+
title={OMAR-RQ: Open Music Audio Representation Model Trained with Multi-Feature Masked Token Prediction},
|
| 73 |
+
author={Fust, Albert and Pons, Jordi and Bogdanov, Dmitry and O帽oro-Rubio, Daniel and G贸mez, Emilia},
|
| 74 |
+
journal={arXiv preprint arXiv:2507.03482},
|
| 75 |
+
year={2025}
|
| 76 |
+
}
|
| 77 |
+
```
|