Multilingual Representation Distillation with Contrastive Learning

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1477–1490 May 2-6, 2023 ©2023 Association for Computational Linguistics Multilingual Representation Distillation with Contrastive Learning Weiting Tan♠⇤ Kevin Heffernan~ Holger Schwenk~ Philipp Koehn♠~ ♠Johns Hopkins University ~Meta AI Research {wtan12, phi}@jhu.edu {kevinheffernan,schwenk}@meta.com Abstract Multilingual sentence representations from large models encode semantic information from two or more languages and can be used for different cross-lingual information retrieval and matching tasks. In this paper, we integrate contrastive learning into multilingual represen- tation distillation and use it for quality estima- tion of parallel sentences (i.e., find semantically similar sentences that can be used as transla- tions of each other). We validate our approach with multilingual similarity search and corpus filtering tasks. Experiments across different low-resource languages show that our method greatly outperforms previous sentence encoders such as LASER, LASER3, and LaBSE. 1 Introduction With the rise of neural networks, high-dimensional word-level sequence representations play an impor- tant role in almost any natural language processing task. Contextual representations from large pre- trained language models (Vaswani et al., 2017; De- vlin et al., 2019; Liu et al., 2019; CONNEAU and Lample, 2019) have shown advantages compared to earlier static embeddings (Mikolov et al., 2013; Pennington et al., 2014). However, they are not pre-trained with sentence-level objectives and their representations of two different sentences cannot be easily used to indicate semantic similarity. To en- code sentence-level information, LASER (Artetxe and Schwenk, 2019b) pools a sentence embedding from the encoder and feeds it to the decoder. An- other approach is to use siamese-structured models where two identical encoders are used to repre- sent sentences of similar meaning and are trained to push their representations close to each other. Sentence-Transformers (Reimers and Gurevych, 2019) are siamese-structured models that are ini- tialized with pre-trained large models like BERT ⇤ Work was done during an internship at Meta AI Re- search. student encoder teacher encoder cosine loss sentence embedding sentence embedding gradient "bring them here to me, " he said. target na ka mea ia, mauria mai ki konei ki ahau. source Figure 1: Student-Teacher Distillation in LASER3. (Devlin et al., 2019) or Roberta (Liu et al., 2019). After fine-tuning, Sentence-Transformers improve their sentence representation for cross-lingual tasks. Besides fine-tuning with identical (siamese) en- coders, distillation can be used to retrieve better representations. Reimers and Gurevych (2020a) extends a monolingual sentence representation into a multilingual representation with model distilla- tion. Similarly, Heffernan et al. (2022) proposed LASER3, a student-teacher architecture that distills information from a pre-trained teacher encoder to a student encoder in new languages. As shown in Figure 1, the distillation process updates the stu- dent encoder with the gradient from the cosine loss. Note that it freezes the parameters of the teacher encoder, which is already pre-trained on the tar- get language (English in our case). Therefore, the target sentence embedding is fixed and the corre- sponding source embedding is aligned with the target embedding. In this paper, we focus on the …

Content truncated to 500 words