lightonai/GTE-ModernColBERT-v1 · Hugging Face

GTE-ModernColBERT has been trained with knowledge distillation on MS MARCO with a document length of 300 tokens, explaining its default value for documents length.

However, as illustrated in the ModernBERT paper, ColBERT models can generalize to documents lengths way beyond their training length and GTE-ModernColBERT actually yields results way above SOTA in long-context embedding benchmarks, see LongEmbed results.

GTE-ModernColBERT is the first model to outpeform ColBERT-small on the BEIR benchmark. As reproduction in the IR domain is challenging, we worked closely with Benjamin Clavié, the author of ColBERT-small to reproduce the evaluation setup of this model. Despite all these efforts and reducing to the maximum the difference in scores in most of the datasets, some are still a bit different. For this reason, we also report the results of ColBERT-small in the same setup we used to evaluate GTE-ModernColBERT for completness and fair comparison.

GTE-ModernColBERT has been trained with knowledge distillation on MS MARCO with a document length of 300 tokens, explaining its default value for documents length. However, as illustrated in the ModernBERT paper, ColBERT models can generalize to documents lengths way beyond their training length and GTE-ModernColBERT actually yields results way above SOTA (almost 10 points above previous SOTA) in long-context embedding benchmark: