arxiv:2506.15154

SonicVerse: Multi-Task Learning for Music Feature-Informed Captioning

Published on Jun 18

· Submitted by

dorienh on Jun 20

Upvote

Authors:

Anuradha Chopra ,

Dorien Herremans

Abstract

SonicVerse, a multi-task music captioning model, integrates audio feature detection to enhance caption quality and enable detailed descriptions of music pieces.

AI-generated summary

Detailed captions that accurately reflect the characteristics of a music piece can enrich music databases and drive forward research in music AI. This paper introduces a multi-task music captioning model, SonicVerse, that integrates caption generation with auxiliary music feature detection tasks such as key detection, vocals detection, and more, so as to directly capture both low-level acoustic details as well as high-level musical attributes. The key contribution is a projection-based architecture that transforms audio input into language tokens, while simultaneously detecting music features through dedicated auxiliary heads. The outputs of these heads are also projected into language tokens, to enhance the captioning input. This framework not only produces rich, descriptive captions for short music fragments but also directly enables the generation of detailed time-informed descriptions for longer music pieces, by chaining the outputs using a large-language model. To train the model, we extended the MusicBench dataset by annotating it with music features using MIRFLEX, a modular music feature extractor, resulting in paired audio, captions and music feature data. Experimental results show that incorporating features in this way improves the quality and detail of the generated captions.

View arXiv page View PDF Add to collection

Community

dorienh

Paper author Paper submitter 1 day ago

Try out the demo for this paper at https://huggingface.co/spaces/amaai-lab/SonicVerse

dorienh

Paper author Paper submitter 1 day ago

SonicVerse is a multi-task music captioning model that integrates caption generation with auxiliary music feature detection tasks such as key detection, vocals detection, and more. The model directly captures both low-level acoustic details as well as high-level musical attributes through a novel projection-based architecture that transforms audio input into natural language captions while simultaneously detecting music features through dedicated auxiliary heads. Additionally, SonicVerse enables the generation of temporally informed long captions for extended music pieces by chaining outputs from short segments using large language models, providing detailed time-informed descriptions that capture the evolving musical narrative.