Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4512–4525, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4512 Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation Nils Reimers and Iryna Gurevych Ubiquitous Knowledge Processing Lab (UKP-TUDA) Department of Computer Science, Technische Universit¨at Darmstadt www.ukp.tu-darmstadt.de Abstract We present an easy and efficient method to ex- tend existing sentence embedding models to new languages. This allows to create multi- lingual versions from previously monolingual models. The training is based on the idea that a translated sentence should be mapped to the same location in the vector space as the orig- inal sentence. We use the original (monolin- gual) model to generate sentence embeddings for the source language and then train a new system on translated sentences to mimic the original model. Compared to other methods for training multilingual sentence embeddings, this approach has several advantages: It is easy to extend existing models with relatively few samples to new languages, it is easier to en- sure desired properties for the vector space, and the hardware requirements for training are lower. We demonstrate the effectiveness of our approach for 50+ languages from various lan- guage families. Code to extend sentence em- beddings models to more than 400 languages is publicly available.1 1 Introduction Mapping sentences or short text paragraphs to a dense vector space, such that similar sentences are close, has wide applications in NLP. However, most existing sentence embeddings models are monolin- gual, usually only for English, as applicable train- ing data for other languages is scarce. For multi- and cross-lingual scenarios, only few sentence em- beddings models exist. In this publication, we present a new method that allows us to extend existing sentence embeddings models to new languages. We require a teacher model M for source language s and a set of parallel (translated) sentences ((s1, t1), ..., (sn, tn)) with ti the translation of si. Note, the ti can be in different 1Code, models, and datasets: https://github.com/ UKPLab/sentence-transformers languages. We train a new student model ˆ M such that ˆ M(si) ≈ M(si) and ˆ M(ti) ≈ M(si) using mean squared loss. We call this approach multi- lingual knowledge distillation, as the student ˆ M distills the knowledge of the teacher M in a mul- tilingual setup. We demonstrate that this type of training works for various language combinations as well as for multilingual setups. We observe an especially high improvement of up to 40 accu- racy points for low resource languages compared to LASER (Artetxe and Schwenk, 2019b). The student model ˆ M learns a multilingual sen- tence embedding space with two important proper- ties: 1) Vector spaces are aligned across languages, i.e., identical sentences in different languages are close, 2) vector space properties in the original source language from the teacher model M are adopted and transferred to other languages. The presented approach has various advantages compared to other training approaches for multilin- gual sentence embeddings. LASER (Artetxe and Schwenk, 2019b) trains an encoder-decoder …

Content truncated to 500 words