Cross-lingual Sentence Embedding using Multi-Task Learning

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9099–9113 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9099 Cross-lingual Sentence Embedding using Multi-Task Learning Koustava Goswami1∗, Sourav Dutta2, Haytham Assem2, Theodorus Fransen1 and John P. McCrae1 1 Data Science Institute, National University of Ireland Galway, Ireland 2 Huawei Research Centre, Dublin, Ireland {koustava.goswami,theodorus.fransen, john.mccrae}@insight-centre.org {sourav.dutta2, haytham.assem}@huawei.com Abstract Multilingual sentence embeddings capture rich semantic information not only for mea- suring similarity between texts but also for catering to a broad range of downstream cross- lingual NLP tasks. State-of-the-art multilin- gual sentence embedding models require large parallel corpora to learn efficiently, which con- fines the scope of these models. In this paper, we propose a novel sentence embedding frame- work based on an unsupervised loss function for generating effective multilingual sentence embeddings, eliminating the need for parallel corpora. We capture semantic similarity and relatedness between sentences using a multi- task loss function for training a dual encoder model mapping different languages onto the same vector space. We demonstrate the effi- cacy of an unsupervised as well as a weakly supervised variant of our framework on STS, BUCC and Tatoeba benchmark tasks. The proposed unsupervised sentence embedding framework outperforms even supervised state- of-the-art methods for certain under-resourced languages on the Tatoeba dataset and on a monolingual benchmark. Further, we show enhanced zero-shot learning capabilities for more than 30 languages, with the model be- ing trained on only 13 languages. Our model can be extended to a wide range of languages from any language family, as it overcomes the requirement of parallel corpora for training. 1 Introduction Sentence embeddings provide an efficient way to encode semantic information of text by mapping texts onto a shared vector space, such that sentences with similar meaning are represented by similar representations. With the abundance of data in diverse languages, cross-lingual sentence embed- ding enable the mapping of multilingual texts into a single unified vector space for a wide range of Natural Language Processing (NLP) tasks. Current ∗Work started during internship at Huawei Research. sentence embedding methods are predominantly monolingual systems, geared mainly towards En- glish (Conneau et al., 2017; Yin et al., 2020). While there exist multilingual sentence embedding frame- works, they are mostly supervised methods requir- ing a large parallel corpus for training. For under- resourced languages, there is not sufficient training data to effectively learn a model and we show that our unsupervised approach can better exploit the available unsupervised data, and thus produce bet- ter results for under-resourced languages. This is achieved by using a dual-encoder architecture based on word-level semantic similarity score (via Word Mover’s Distance) and learning to embed this into a single vector for sentences. Supervised sentence embedding approaches map parallel sentences from source and target languages into the same vector space by either maximising their cosine similarity or minimising the distance between the generated embeddings (Artetxe and Schwenk, 2019; Reimers and Gurevych, 2020). For example, recent supervised methods using parallel corpus rely on a teacher-student …

Content truncated to 500 words