A Multi-Pairwise Extension of Procrustes Analysis for Multilingual Word Translation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3560–3565, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3560 A Multi-Pairwise Extension of Procrustes Analysis for Multilingual Word Translation Hagai Taitelbaum Faculty of Engineering Bar-Ilan University, Israel hagait62@gmail.com Gal Chechik The Gonda Brain Research Center Bar-Ilan University, NVIDIA gal.chechik@biu.ac.il Jacob Goldberger Faculty of Engineering Bar-Ilan University, Israel jacob.goldberger@biu.ac.il Abstract In this paper we present a novel approach to si- multaneously representing multiple languages in a common space. Procrustes Analysis (PA) is commonly used to find the optimal orthog- onal word mapping in the bilingual case. The proposed Multi Pairwise Procrustes Analysis (MPPA) is a natural extension of the PA algo- rithm to multilingual word mapping. Unlike previous PA extensions that require a k-way dictionary, this approach requires only pair- wise bilingual dictionaries that are much easier to construct in either a supervised or an unsu- pervised way. The improved performance of the MPPA algorithm is demonstrated on two standard multilingual tasks. 1 Introduction Continuous word embeddings have been proved effective in numerous NLP applications. In cross- language tasks, these vector-space representations have recently emerged as a tool to transfer knowl- edge from one language to another. Specifically, several studies have suggested forming cross- lingual embeddings by learning a linear mapping from a source-language embedding space to a target-language one and demonstrated the bene- fits of this approach for word translation (Mikolov et al., 2013; Klementiev et al., 2012). Xing et al. (2015) showed that imposing orthogonality con- straints on the linear mapping between spaces can alleviate overfitting. Building on these concepts, several studies have aimed to improve these bilin- gual word embeddings using bilingual word dic- tionaries that are created in either a supervised or an unsupervised manner (Artetxe et al., 2017a). Bilingual word embedding were found to be use- ful in a number of monolingual and cross-lingual NLP tasks (Vulic and Moens, 2015; Tsai and Roth, 2016). Bilingual embedding can be extended to a mul- tilingual setup by jointly learning mappings from each monolingual word embedding to a shared vector space. Modeling multiple languages jointly has been shown to improve modeling accuracy on bilingual tasks because it can utilize knowledge learned from the other languages (Ammar et al., 2016; Duong et al., 2017; Taitelbaum et al., 2019). Extending the bilingual setup to a multilingual setting poses new challenges. For bilingual em- bedding, the word-mapping problem has a closed- form solution known as Orthogonal Procrustes Analysis (PA), which can be computed using sin- gular value decomposition (Schnemann, 1966). However, there is no similar closed-form solution for the multi-language case. The standard exten- sion of PA to multi-set alignment is Generalized Procrustes Analysis (GPA) (Gower, 1975) which is an iterative greedy algorithm. GPA was recently used to jointly transform multiple languages into a shared vector space (Kementchedjhieva et al., 2018). However, GPA assumes that a multi-way word correspondence is available, which is often not …

Content truncated to 500 words