Factorized MLLM Embeddings for Composed Image Retrieval

FactEmbed decouples composed-query comprehension from embedding learning by training factorized MLLM latents with target imagination, factor-level matching, and attention distillation, then reranking with mean-pooled latents.

Composed Image Retrieval retrieves a target image given a reference image and a modification text. Recent MLLM approaches finetune contrastively by compressing the composed query and target into a single embedding, typically from the final EOS token. This direct mapping provides no explicit signal for the MLLM to resolve the requested visual edit before forming a discriminative embedding. The paper argues that the MLLM's compositional understanding therefore remains underutilized.

FactEmbed prepends K learnable factorized latents before the EOS token in both query and target sequences, using them as information bottlenecks. A lightweight decoder attached at an intermediate MLLM layer reconstructs target patch features from these latents, conditioned on a coarse target layout prior, and produces normalized alpha masks over spatial positions. The latents are aligned for retrieval with a factor-level InfoNCE using Sinkhorn optimal transport, while the EOS hidden state is optimized with an instance-level InfoNCE as the global embedding. The decoder's alpha masks are distilled into the latents' attention at a deeper layer via KL divergence. Training is staged: reconstruction-only warmup, then joint MLLM, latent, and decoder training with DoRA and the full objective, then post-training with MLLM-augmented triplets. At inference, EOS retrieves candidates and mean-pooled latents rerank the top-100 shortlist.

• FactEmbed improves Recall@1 on Refined-CIRR over CSMCIR and ReCALL, and average Recall@10 on Refined-FashionIQ over the same baselines, by the stated percentages. §4.1
• On the CIRR test set, FactEmbed reports Recall@1 56.60, Recall@5 84.87, Recall@10 91.78, subset Recall@1 81.86, and Avg 83.37. §4.1
• On Refined-CIRR, FactEmbed reports Recall@1 68.99, subset Recall@1 87.45, and Avg 78.22; on Refined-FashionIQ it reports average Recall@10 71.13 and average Recall@50 86.41. §4.1
• Efficient reranking narrows the R@1 gap to Full Factor to 0.24 and 0.21 while using only two vectors per gallery image. §4.2
• The full model after post-training surpasses the post-trained baseline by 2.91 and 2.57 in R@1 on CIRR and Refined-CIRR, and by 2.87 and 2.37 in R@10 on FashionIQ and Refined-FashionIQ. §4.3
• Changing the backbone from Qwen2.5-VL-7B to Qwen3-VL-8B improves R@1 by 2.65 points on the CIRR test set, versus 1.60 points reported by ReCALL for the same transition. §4.4

• A fixed K ignores input complexity because no explicit supervision specifies the number of semantic factors; query-adaptive latents and matching schemes across varying set sizes remain future work.
• Full-factor retrieval achieves higher accuracy, but mean-pooling the latents for efficient reranking leaves part of the fine-grained information unexploited.
• Evaluation is limited to CIRR and FashionIQ, so generalizability as a unified multimodal retriever requires further verification.
• Contrastive captioning conditions each rewrite only on the original target, not among multiple hard negatives for the same query; as more hard negatives are mined, sibling hard negatives may overlap and introduce false negatives.

• A set of learnable factorized latents is prepended before the EOS token in query and target sequences; the latents interact with multimodal inputs through causal attention and act as information bottlenecks. §3.1
• Target imagination uses a lightweight decoder with a self-attention layer and 3-layer MLP attached to an intermediate MLLM layer to reconstruct target patch representations from the latents. §3.1
• The decoder predicts spatial logits and reconstruction features per latent; softmax over the latent dimension yields alpha masks, and the reconstruction is a mask-weighted sum supervised by MSE against target features. §3.1
• Factor-level matching applies InfoNCE to latents using Sinkhorn optimal transport over pairwise cosine similarities, while instance-level matching applies the same contrastive loss to the EOS global embedding. §3.2
• A KL distillation loss aligns the deeper-layer attention distributions of the factorized latents with the decoder's spatial alpha masks; the spatial distillation loss is formulated only on the target-side forward pass. §3.2
• Training uses a staged strategy: Stage 1 optimizes only reconstruction to warm up latents and decoder, Stage 2 trains the MLLM backbone with DoRA and the full objective, and post-training adds MLLM-augmented triplets. §3.3
• Implementation details set K=8 latents, a 2x2 target layout prior, decoder layer ℓ=14, and distillation from the third layer from the top with loss coefficients λ_imag=1.0 and λ_distill=0.5. §4
• The diversity analysis reports lower within-instance latent similarity for the full method than for latents trained with factor-level InfoNCE alone, including after prototype removal. §B