English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddings

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9122 - 9133 December 7-11, 2022 ©2022 Association for Computational Linguistics English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddings Yau-Shian Wang Ashley Wu Graham Neubig Carnegie Mellon University king6101@gmail.com wangchew@andrew.cmu.edu gneubig@cs.cmu.edu Abstract Universal cross-lingual sentence embeddings map semantically similar cross-lingual sen- tences into a shared embedding space. Align- ing cross-lingual sentence embeddings usually requires supervised cross-lingual parallel sen- tences. In this work, we propose mSimCSE, which extends SimCSE (Gao et al., 2021) to multilingual settings and reveal that contrastive learning on English data can surprisingly learn high-quality universal cross-lingual sentence embeddings without any parallel data. In unsu- pervised and weakly supervised settings, mSim- CSE significantly improves previous sentence embedding methods on cross-lingual retrieval and multilingual STS tasks. The performance of unsupervised mSimCSE is comparable to fully supervised methods in retrieving low- resource languages and multilingual STS. The performance can be further enhanced when cross-lingual NLI data is available. 1 1 Introduction Universal cross-lingual sentence embeddings map the sentences from multiple languages into a shared embedding space, where semantically sim- ilar sentences across languages are close to each other. These embeddings have a wide spectrum of applications such as multi-lingual document re- trieval (Artetxe and Schwenk, 2019a; Lin et al., 2020), multi-lingual question answering (Asai et al., 2021a,b; Kumar et al., 2022), unsupervised machine translation (Tran et al., 2020), and zero- shot transfer learning (Phang et al., 2020). As shown in Figure 1 (a), without finetuning on downstream tasks, the embedding space of pre- trained multilingual language models such as m- BERT (Devlin et al., 2019) or XLM-R (Conneau et al., 2020) separate the embeddings of each lan- guage into different clusters. To align cross-lingual 1Our code is publicly available at https://github. com/yaushian/mSimCSE. (a) XLM-R without finetun- ing. (b) XLM-R fintuned on En- glish NLI data. Figure 1: We visualize the sentence embeddings on XNLI corpus, where blue dots and green dots denote the sentences from English and Swahili respectively. Here, red dots, black dots, and purple dots denote the parallel sentences from different languages. In (a), the sentence embeddings from different languages are clearly sepa- rated into two clusters. In (b), after English NLI training, the embedding space becomes indistinguishable for dif- ferent languages, and the parallel sentences are aligned to each other. sentence embeddings, previous work (Artetxe and Schwenk, 2019a; Chidambaram et al., 2019; Feng et al., 2020) finetunes multilingual language mod- els with billions of parallel data. However, it is non-trivial to obtain numerous parallel data for all languages. One potential direction to alleviate the need for parallel data is to enhance cross-lingual transfer of sentence embeddings. Pre-trained multilingual language models (Pires et al., 2019; Phang et al., 2020) have shown im- pressive performance on cross-lingual zero-shot transfer (Pires et al., 2019) that a model finetuned on a source language can generalize to target lan- guages. This implies the representations finetuned on downstream tasks are universal across various languages. In this work, we explore …

Content truncated to 500 words