arxiv:2509.02108

DivMerge: A divergence-based model merging method for multi-tasking

Published on Sep 2

· Submitted by

Gwénolé Lecorvé on Sep 9

Upvote

Authors:

Fosse Loïc ,

Abstract

Multi-task learning (MTL) is often achieved by merging datasets before fine-tuning, but the growing availability of fine-tuned models has led to new approaches such as model merging via task arithmetic. A major challenge in this setting is task interference, which worsens as the number of tasks increases. We propose a method that merges models trained on different tasks into a single model, maintaining strong performance across all tasks. Our approach leverages Jensen-Shannon divergence to guide the merging process without requiring additional labelled data, and automatically balances task importance. Unlike existing methods, our approach remains robust as the number of tasks grows and consistently outperforms prior work.

View arXiv page View PDF Add to collection

Community

glecorve

Paper submitter 10 days ago

This paper introduces a new method for model merging method that preserves high performance on each task when increasing the number of merged tasks. The key is the use of Jensen-Shannon divergence to drive the optimization of interpolation weights and to avoid the need for labeled data for this optimization, i.e. automatically balancing task importance. Unlike previous methods, this approach remains robust and effective as the number of tasks increases, consistently outperforming existing techniques.