To Align or Not To Align:

Check Your C🧭MPASS Before You Train

Which models are compatible with each other?

Combining pretrained models is a common strategy for building AI systems, whether by aligning a vision encoder with a language model or merging fine-tuned checkpoints. However, finding compatible pairs typically requires costly empirical search, as compatibility is known only after alignment or merging.

We propose COMPASS (COMPatibility ASsessment Score), a theoretically motivated diagnostic that estimates whether a model pair will be compatible prior to alignment or merging.

Overview of COMPASS. Given a library of candidate models, COMPASS estimates pair compatibility from local neighborhood agreement before alignment or merging. COMPASS decomposes compatibility into a pair-specific ease score and a paradigm-level ceiling score, then combines them to rank candidate pairs.

Why local agreement?

A growing body of work suggests that models can share meaningful representational structure even when their global geometries differ. The Platonic Representation Hypothesis proposed that sufficiently expressive models converge toward a shared global representation, whereas the Aristotelian Representation Hypothesis showed that this agreement is mainly local: models trained with different objectives and data often agree on who is close to whom, even when their global geometries differ.

This makes local neighborhood overlap a natural candidate for a compatibility diagnostic. It is cheap to measure, requiring only forward passes over a small probe set, and can be computed before alignment or merging.

We go beyond this empirical observation by providing a theoretical motivation for why local overlap should be informative about model-pair compatibility.

Two axes of compatibility

COMPASS separates model-pair compatibility into two complementary components that capture different limitations on successful alignment or merging.

Ease

How easy is it to make this specific pair compatible?

Ease is pair-specific. It measures local neighborhood agreement at the start of the relevant dynamics: before cross-modal alignment, or between the two fine-tuned checkpoints before merging.

Pairs that already agree on local neighborhoods are easier to make compatible within a fixed training budget.

Ceiling

What level of compatibility can this paradigm support?

Ceiling is paradigm-level. It estimates how much local agreement the corresponding training paradigms can support on average.

Even when a specific pair begins favorably, differences between the objectives that shaped the two representations can limit the agreement that is ultimately achievable.

The continuous COMPASS score provides the ranking. High ease means the specific pair already agrees locally, while high ceiling means the training paradigms support a high level of compatibility. Together, the two axes distinguish pairs that are ready to combine from those limited by optimization, paradigm, or both.

COMPASS predicts downstream compatibility

We evaluate COMPASS in two applications: cross-modal alignment across 72 vision-language pairs and task-arithmetic model merging across 392 pairwise merges.

In both settings, the COMPASS score strongly predicts the outcome obtained after actually aligning or merging the models.

Across cross-modal alignment, Ease × Ceiling reaches a Pearson correlation of 0.90 with Flickr30k image-to-text R@1 and 0.91 with COCO image-to-text R@1. For task-arithmetic merging, COMPASS reaches a Pearson correlation of 0.70 with mean retention. The product of ease and ceiling is more predictive than either component alone.

COMPASS is built on local k-NN overlap, rather than global representational similarity. Global CKA shows lower correlation with downstream outcomes than COMPASS does. On Flickr30k and COCO alignment, global CKA reaches correlations of 0.63/0.68, compared with 0.90/0.91 for COMPASS; for merging, global CKA reaches 0.25, compared with 0.70 for COMPASS.

Cross-modal alignment

Cross-modal alignment trains a learned adapter on paired image-text data so that a frozen vision encoder and a frozen language model produce compatible representations. Given a library of possible vision and language models, the practical question is therefore: which pair should we align?

We evaluate 72 vision-language pairs spanning 19 vision models and 10 text encoders. For alignment, ease is the pretrained cross-modal k-NN overlap of the specific vision-language pair, while ceiling measures the paradigm-level agreement supported by the vision encoder relative to the language-model library.

Contrastive vision-language paradigms such as CLIP and SigLIP tend to have high ceiling, while paradigms whose objectives are less closely tied to language, such as masked reconstruction, show lower ceiling.

The phase diagram separates the candidate pairs into the four COMPASS regimes. Measuring downstream success on Flickr30k, act now pairs achieve the highest mean retrieval performance, followed by ceiling-limited, budget-limited, and incompatible pairs. The same ordering holds for downstream success on COCO.

Model-merging

We further apply COMPASS to the task of combining models directly in weight space. More specifically, we study task arithmetic, where two checkpoints of the same base model (each fine-tuned on a different task) are averaged with the goal of retaining both capabilities.

We evaluate 392 pairwise merges across 14 pretrained backbones and 8 image-classification tasks.

Here, ease measures the pre-merge neighborhood agreement between the two fine-tuned checkpoints. Ceiling measures the specialization of their pretrained base model relative to the reference vision-model library.

The same two-axis diagnostic separates successful from unsuccessful merges. Act-now pairs achieve the highest mean retention, while incompatible pairs perform substantially worse.

Interestingly, budget-limited and ceiling-limited pairs can have similar average merge quality while calling for different actions. The regime therefore adds information that the continuous score alone does not provide.

Score models once, reuse the representations

Exhaustive downstream evaluation scales with the number of candidate pairs. COMPASS instead extracts representations from each model once and reuses them across pairwise comparisons.

For a 19 × 10 alignment grid, COMPASS requires 29 probe passes instead of 190 downstream alignment runs.

In our experimental setup, this corresponds to 21 minutes for COMPASS compared with 86-102 hours for exhaustive downstream evaluation, more than two orders of magnitude less wall-clock time.

Publication

To Align or Not To Align: Check Your COMPASS Before You Train
Fabian Gröger*, Petar Damjanović*, Maria Brbić.

Code

An implementation is available on GitHub.

Contributors

The following people contributed to this work:

Fabian Gröger*

Petar Damjanović*

Maria Brbić