[ICLR 2025] ACE-Merging: Solving the Inter-Task Interference Puzzle Without Data

Executive Summary

TL;DR: ACE-Merging is a novel framework for merging specialized neural networks into a single multi-task model without needing access to any training data. By theoretically proving that a task's "input covariance" is hidden within its fine-tuned weights, the authors provide a principled closed-form solution that significantly outperforms existing heuristic methods like Task Arithmetic or Ties-Merging.

Academic Context: This work represents a shift from heuristic-based parameter manipulation to optimization-grounded merging. It addresses the "interference" problem—where tasks overwrite each other's knowledge—by effectively reconstructing the geometry of the task space.

The Core Problem: Blind Merging

Most modern model merging techniques (e.g., Task Arithmetic) treat parameters like simple vectors in Euclidean space. However, neural network weights aren't just numbers; they represent transformations of specific data distributions.

When we merge Task A (trained on medical text) and Task B (trained on Python code), they often interfere because simple averaging doesn't know which weights are critical for which distribution. Prior attempts to solve this involved:

1. Data-dependent merging: Requires the original data (often private/unavailable).
2. Iterative optimization: Slow and sensitive to hyperparameters.
3. Heuristics: Pruning or sign-flipping (e.g., TIES), which solve the symptoms rather than the cause.

Methodology: The "Hidden" Covariance

The authors' breakthrough is Theorem 1: The input covariance of a task is proportional to the covariance of its weight displacements ($\Delta W$). This means the fine-tuning process itself leaves a "fingerprint" of the data distribution in the weights.

The ACE-Merging Pipeline

The algorithm operates in three distinct stages to ensure a stable and accurate fusion:

1. Adaptive Covariance Normalization: It calculates a heterogeneity metric ($\gamma$). If tasks vary wildly in energy (e.g., one task had a high learning rate), it normalizes them to prevent one task from dominating the merged model.
2. Collective Structural Prior (CSP): It uses shared feature geometry across all tasks as an "informative prior," ensuring that dimensions of collective importance are amplified.
3. Spectral Refinement: Perhaps the most vital step. The initial closed-form solution often becomes "ill-conditioned," where a few dimensions hold all the energy (see Figure 2). Spectral Refinement identifies the correct "subspace" and flattens the energy distribution to make the model robust.

Figure 2: The left panel shows how the raw merged model is spectrally "peaky" (unstable), while the authors' refinement (right) restores a healthy, expressive energy profile.

Experimental Performance: Setting a New SOTA

ACE-Merging was tested across a massive array of benchmarks, including GPT-2, RoBERTa, and Vision Transformers (ViT).

In the vision domain, accurately merging up to 20 tasks into a single ViT-L/14 model showed that the performance gap between ACE-Merging and previous methods grows as the problem gets harder.

Table 1: Performance across ViT scales. Note the consistent leadership of ACE-Merging in the 20-task setting.

Insights & Critical Analysis

Why does this work so much better?

• Isotropic vs. Anisotropic: Traditional averaging is "isotropic"—it treats every direction in the weight space as equally important. ACE-Merging is "anisotropic"—it understands the specific directions (eigenvectors) that matter for each task.
• The Power of the Closed Form: Unlike gradient-based methods (which can get stuck in local minima), ACE-Merging provides a direct mathematical solution. This makes it faster and more stable.

Limitations: The method relies on a global regularization strength ($ϵ$). While the authors show it's robust, an "automated" way to estimate $ϵ$ per task would be the final piece of the puzzle for truly autonomous model merging.

Conclusion

ACE-Merging bridges the gap between theoretical machine learning and practical model deployment. By treating model merging as an estimation problem rather than a simple math operation, it enables the creation of powerful, multi-talented models from specialized individuals—completely data-free.