Rethinking Projector Training in Multimodal LLMs

Introduction

Multimodal LLMs are typically trained in two stages. First, a projector is trained on large image-caption datasets to align a frozen vision encoder with a frozen language model. This is followed by joint training of the projector and language model on multimodal instructions. Despite its widespread use, the role of the projector alignment stage remains poorly understood.

In this work, we show that projector alignment primarily performs coarse alignment, mapping visual features into a language-model-compatible input space that admits a large set of possible solutions. Motivated by these findings, we propose PORTAL (Projector via Optimal-transpoRT ALignment), which constructs the projector directly from the marginal feature statistics of the two modalities, eliminating the need for paired image-caption data or gradient-based optimization during projector alignment.

Figure 1. Projector alignment as coarse alignment with a broad solution space. Left: Standard two-stage pipeline, where only the projector is trained during initial alignment. Middle: Projectors trained on different datasets from a shared initialization can be linearly interpolated while preserving downstream performance. Right: This motivates PORTAL, a closed-form projector constructed from the marginal statistics of visual and language features using optimal transport.

Projector alignment saturates with a small number of samples

To understand what projector alignment actually learns, we first examine how downstream performance scales with the amount of Stage-I training data. We vary the amount of paired image-caption data used for projector alignment while keeping the subsequent multimodal instruction tuning unchanged, and evaluate performance across 16 benchmarks covering Knowledge, General, OCR, and Vision-Centric tasks.

We find that using only 10–20% of the LCS-558k image-caption pairs already recovers most downstream performance gains, with diminishing improvements as more data is added. We observe a similar trend on the larger ShareGPT-1.2M dataset. OCR and Chart tasks benefit the most from projector alignment, while Knowledge and Vision-Centric performance saturates earlier.

These results suggest that projector alignment is not primarily learning fine-grained visual-language correspondences. Rather, it appears to perform a coarse-grained alignment step, placing visual features into a compatible region of the LLM input space from which subsequent joint training can effectively adapt the full multimodal model.

Figure 2. Projector alignment saturates early with limited training data. Relative downstream performance gains over the no-pretraining baseline as a function of Stage-I data fraction, evaluated on LCS-558k (left) and ShareGPT-1.2M (right). Most gains are recovered with a small fraction of the training data, while OCR and Chart tasks show the largest benefits from paired alignment.

Figure 3. Projectors trained on different datasets are linearly interpolable. Interpolating between projectors trained on LCS-558k and ShareGPT-1.2M from a shared initialization largely preserves downstream performance. The equal-weight mixture reaches the highest aggregate score of 49.0, outperforming either trained endpoint.

PORTAL: Training-free projector via optimal transport

Our findings suggest that projector alignment primarily serves as geometric conditioning, rather than requiring a precisely learned correspondence between individual images and captions. This motivates a simpler alternative to conventional projector training.

We introduce PORTAL, a training-free method that constructs the projector from the marginal feature statistics of the vision encoder and language model. PORTAL approximates each modality’s feature distribution with a Gaussian, using its mean, principal directions, and spread along each direction. It then computes a closed-form optimal-transport map between the two Gaussian approximations in a shared principal subspace.

Intuitively, PORTAL first normalizes the geometry of visual features, then maps and rescales their dominant directions to match the geometry of the language model’s input embeddings. This provides a compatible projector initialization without learning from paired image-caption examples.

In practice, PORTAL requires only a small pool of unlabeled images to estimate vision-side statistics and the LLM’s input embedding table to estimate language-side statistics. A single PCA on each side is sufficient, and the resulting map initializes the standard projector architecture. No paired captions or gradient updates are required for this initialization, while the subsequent multimodal instruction-tuning stage remains unchanged.

Details + guarantees: see Section 4 of our paper.

PORTAL matches trained projector alignment without paired data

We evaluate PORTAL across six LLM backbones, two vision encoders, and 16 benchmarks, covering seven vision encoder–LLM configurations. We compare PORTAL against conventional projector alignment trained on 558k image-caption pairs and a randomly initialized projector that skips the alignment stage. All configurations undergo identical Stage-II multimodal instruction tuning.

Despite requiring no paired data or optimization during projector initialization, PORTAL achieves comparable downstream performance to the standard trained pipeline. It outperforms the trained baseline in three of seven configurations, reaching up to 106.2% of its performance, and retains between 96.3% and 99.3% in the remaining four.

The advantage is particularly clear when random initialization performs poorly. For example, with Vicuna 1.5 7B and SigLIP, random initialization reaches only 82.5% of the trained baseline, whereas PORTAL recovers 99.3%.

These results demonstrate that much of the benefit of projector alignment can be obtained by matching feature distributions, without requiring a uniquely learned image-text correspondence. Some backbone-specific performance gaps remain, particularly on OCR and Chart benchmarks, where fine-grained visual information can benefit from paired supervision.

Table 1. PORTAL generalizes across vision encoders and LLM backbones. Comparison of conventionally trained projectors, random initialization, and PORTAL across seven configurations and 16 benchmarks. PORTAL uses no Stage-I paired captions or gradient updates, while all methods share identical Stage-II instruction tuning. The Avg column reports the mean benchmark score, and %BL gives performance relative to the conventionally trained baseline.

PORTAL reduces the projector alignment budget

Beyond replacing projector alignment entirely, PORTAL can also serve as a strong initialization when paired image-caption data is available. We investigate whether starting from PORTAL can reduce the amount of data and computation needed for subsequent projector training.

On Vicuna 1.5 7B with CLIP, PORTAL consistently outperforms random initialization across different projector alignment budgets. Remarkably, it reaches the performance of the conventional baseline trained on all 558k image-caption pairs with only approximately 4k paired captions, corresponding to 135× fewer projector-alignment FLOPs.

With as few as 10k paired captions, PORTAL already exceeds the fully trained baseline. This shows that PORTAL not only eliminates the need for paired alignment training in many settings, but can also substantially accelerate it when additional optimization is desired.

Figure 4. PORTAL substantially reduces projector alignment compute. Average performance across 16 benchmarks as a function of the Stage-I training budget on Vicuna 1.5 7B/CLIP. PORTAL provides a stronger starting point than random initialization and matches the full 558k-trained baseline with approximately 4k paired captions, requiring 135× fewer alignment FLOPs. Stage-II instruction tuning is identical across all settings.

Takeaway

Our results suggest that projector training in multimodal LLMs primarily performs coarse alignment, placing visual features into a compatible region of the language model’s input space rather than learning a unique, fine-grained visual-language mapping.

Building on this insight, PORTAL replaces or substantially reduces projector alignment through lightweight feature-distribution matching, requiring neither paired captions nor gradient-based optimization for its initialization. By preserving downstream performance while substantially reducing the alignment budget, PORTAL offers a simpler and more efficient approach to constructing multimodal LLMs.

Publication

Rethinking Projector Training in Multimodal LLMs
Fabian Gröger*, Naël Ouerghemi*, Shuo Wen*, Maria Brbić.

Code

An implementation is available on GitHub.

Contributors

The following people contributed to this work:

Fabian Gröger*

Naël Ouerghemi*

Shuo Wen*

Maria Brbić