Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12044–12066 July 9-14, 2023 ©2023 Association for Computational Linguistics Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval John Wieting1, Jonathan H. Clark1, William W. Cohen1, Graham Neubig2, and Taylor Berg-Kirkpatrick3 1Google DeepMind 2Carnegie Mellon University, Pittsburgh, PA, 15213, USA 3University of California San Diego, San Diego, CA, 92093, USA {jwieting,jhclark,wcohen}@google.com, gneubig@cs.cmu.edu, tberg@eng.ucsd.edu Abstract Contrastive learning has been successfully used for retrieval of semantically aligned sentences, but it often requires large batch sizes and care- fully engineered heuristics to work well. In this paper, we instead propose a generative model for learning multilingual text embed- dings which can be used to retrieve or score sen- tence pairs. Our model operates on parallel data in N languages and, through an approximation we introduce, efficiently encourages source sep- aration in this multilingual setting, separating semantic information that is shared between translations from stylistic or language-specific variation. We show careful large-scale compar- isons between contrastive and generation-based approaches for learning multilingual text em- beddings, a comparison that has not been done to the best of our knowledge despite the pop- ularity of these approaches. We evaluate this method on a suite of tasks including seman- tic similarity, bitext mining, and cross-lingual question retrieval—the last of which we intro- duce in this paper. Overall, our Variational Multilingual Source-Separation Transformer (VMSST) model outperforms both a strong contrastive and generative baseline on these tasks.1 1 Introduction Contrastive learning is the dominant paradigm for learning text representations from parallel text (Her- mann and Blunsom, 2014; Singla et al., 2018; Guo et al., 2018; Wieting et al., 2019; Feng et al., 2022). However, contrastive learning requires strong neg- ative examples in the data and finding these nega- tives can be expensive in terms of compute or man- ual effort. In this paper, we propose a generative2 model for learning multilingual text embeddings 1Code and Flax-based T5X model checkpoint available at https://github.com/google-research/ google-research/tree/master/vmsst. 2We mean generative both in terms of text generation and as a statistical model of the joint probability distribution. which encourages source separation, separating se- mantic information that is shared between transla- tions from stylistic or language-specific variation. We find that by filtering this variation into separate variables, performance of the remaining represen- tations, that encode shared semantic information, increases across all downstream tasks. Through an approximation that greatly reduces the memory footprint of our model, we scale our model and train on 92 languages. We systemat- ically compare our model, the Variational Multi- lingual Source-Separation Transformer (VMSST) to strong contrastive and generative baselines on a suite of tasks including semantic similarity, bitext mining, and question retrieval, which we introduce for the cross-lingual setting, using the same train- ing data and architecture. We show that our model outperforms these models and is also competitive with the state-of-the-art. We analyze VMSST with careful ablations, showing the contribution of each aspect of the model to performance. We also show that …

Content truncated to 500 words