Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3807–3815 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3807 Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast Liang Wang and Wei Zhao and Jingming Liu Yuanfudao AI Lab, Beijing, China {wangliang01,zhaowei01,liujm}@yuanfudao.com Abstract In this paper, we propose to align sentence representations from different languages into a unified embedding space, where semantic similarities (both cross-lingual and monolin- gual) can be computed with a simple dot prod- uct. Pre-trained language models are fine- tuned with the translation ranking task. Exist- ing work (Feng et al., 2020) uses sentences within the same batch as negatives, which can suffer from the issue of easy negatives. We adapt MoCo (He et al., 2020) to further im- prove the quality of alignment. As the ex- perimental results show, the sentence repre- sentations produced by our model achieve the new state-of-the-art on several tasks, including Tatoeba en-zh similarity search (Artetxe and Schwenk, 2019b), BUCC en-zh bitext mining, and semantic textual similarity on 7 datasets. 1 Introduction Pre-trained language models like BERT (Devlin et al., 2019) and GPT (Radford and Narasimhan, 2018) have achieved phenomenal successes on a wide range of NLP tasks. However, sentence repre- sentations for different languages are not very well aligned, even for pre-trained multilingual models such as mBERT (Pires et al., 2019; Wang et al., 2020). This issue is more prominent for language pairs from different families (e.g., English versus Chinese). Also, previous work (Li et al., 2020) has shown that the out-of-box BERT embeddings perform poorly on monolingual semantic textual similarity (STS) tasks. There are two general goals for sentence repre- sentation learning: cross-lingual representations should be aligned, which is a crucial step for tasks like bitext mining (Artetxe and Schwenk, 2019a), unsupervised machine translation (Lample et al., 2018b), and zero-shot cross-lingual transfer (Hu et al., 2020) etc. Another goal is to induce a met- ric space, where semantic similarities can be com- puted with simple functions (e.g., dot product on L2-normalized representations). Translation ranking (Feng et al., 2020; Yang et al., 2020) can serve as a surrogate task to align sentence representations. Intuitively speaking, par- allel sentences should have similar representations and are therefore ranked higher, while non-parallel sentences should have dissimilar representations. Models are typically trained with in-batch nega- tives, which need a large batch size to alleviate the easy negatives issue (Chen et al., 2020a). Feng et al. (2020) use cross-accelerator negative sam- pling to enlarge the batch size to 2048 with 32 TPU cores. Such a solution is hardware-intensive and still struggles to scale. Momentum Contrast (MoCo) (He et al., 2020) decouples the batch size and the number of nega- tives by maintaining a large memory queue and a momentum encoder. MoCo requires that queries and keys lie in a shared input space. In self- supervised vision representation learning, both queries and keys are transformed image patches. However, for translation ranking task, the queries and keys come from different input spaces. In this paper, …

Content truncated to 500 words