Supervised and Nonlinear Alignment of Two Embedding Spaces for Dictionary Induction in Low Resourced Languages

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 823–832, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 823 Supervised and Nonlinear Alignment of Two Embedding Spaces for Dictionary Induction in Low Resourced Languages Masud Moshtaghi Amazon / Manhattan Beach, CA USA mmasud@amazon.com Abstract Enabling cross-lingual NLP tasks by leverag- ing multilingual word embedding has recently attracted much attention. An important moti- vation is to support lower resourced languages, however, most efforts focus on demonstrat- ing the effectiveness of the techniques using embeddings derived from similar languages to English with large parallel content. In this study, we present a noise tolerant piecewise linear technique to learn a non-linear map- ping between two monolingual word embed- ding vector spaces. We evaluate our approach on inferring bilingual dictionaries. We show that our technique outperforms the state of the art in lower resourced settings with an aver- age improvement of 3.7% for precision @10 across 14 mostly low resourced languages. 1 Introduction Aligning two embedding spaces is a key compo- nent in many multilingual tasks in natural lan- guage processing such as unsupervised word and sentence translation (Ruder et al., 2017) and large- scale dictionary creation (Lample et al., 2018). Large scale dictionaries are used in many NLP tasks, for example, web data cleaning. The main feature of the two most successful methods for this task is based on dictionaries (Xu and Koehn, 2017; Koehn et al., 2018). Initial success obtained in aligning two embed- ding spaces has created a flurry of research ef- forts into improving the task (Nakashole, 2018; Doval et al., 2018; Kementchedjhieva et al., 2018; Chen and Cardie, 2018; Joulin et al., 2018; Hoshen and Wolf, 2018; Artetxe et al., 2018), to name a few published in 2018. There are two main ap- proaches for mapping a monolingual word embed- ding space into another (Ruder et al., 2017): 1) Su- pervised methods rely on a few anchor points to learn the matching of two spaces (Lample et al., 2018; Mikolov et al., 2013; Xing et al., 2015; Joulin et al., 2018) and; 2) Unsupervised meth- ods mostly formulate the problem as an adver- sarial optimization problem (Lample et al., 2018; Zhang et al., 2017; Ruder et al., 2017; Chen and Cardie, 2018; Hoshen and Wolf, 2018). While for high density languages such as Spanish, German, French and Italian the accuracy of unsupervised techniques is comparable, or even higher, than su- pervised techniques, the accuracy of unsupervised techniques is significantly lower when lower re- sourced languages are considered. A number of the experiments featured in the recent publications reflect this fact (Søgaard et al., 2018; Irvine and Callison-Burch, 2017). To place this work in the context of recent ad- vances in the field, we divide the aspects targeted by researchers to improve the precision with some examples in Fig. 1. While some of the improve- ments in optimization and space-adjustments can be applied …

Content truncated to 500 words