Finetuning Multi-Vector Embedding Models with Sentence Transformers

Introduction to Multi-Vector Models

The article introduces a method for training and finetuning multi-vector embedding models using the Sentence Transformers library. It highlights the `MultiVectorEncoder`, designed for ColBERT-style late interaction retrieval. This approach enables users to finetune models that can surpass the performance of general-purpose retrievers on specific datasets.

Finetuning Process and Components

Finetuning multi-vector models involves several key components: the model architecture, datasets, loss functions, training arguments, evaluators, and the trainer class. The guide provides practical examples for each component, demonstrating how they are used to achieve strong multi-vector model performance. The entire process runs on `pip install -U "sentence-transformers[train]"`.

Performance and Evaluation

An evaluation section demonstrates the effectiveness of the finetuning method. A finetuned `multi-vector-encoder/mLateOn-medical` model, trained in 14.5 hours on a single RTX 3090, outperformed various general-purpose retrieval models (dense, sparse, lexical, and multi-vector) on a medical retrieval evaluation.

Multi-Vector vs. Dense Embedding Models

Unlike dense embedding models that compress text into a single vector, multi-vector models (also known as late-interaction or ColBERT-style models) retain one small vector per token. This allows for token-level matching, where each query token finds its best-matching document token, and scores are summed. This fine-grained signal preservation typically leads to stronger retrieval, though it may result in a larger index.