A Locally Linear Procedure for Word Translation

Proceedings of the 28th International Conference on Computational Linguistics, pages 6013–6018 Barcelona, Spain (Online), December 8-13, 2020 6013 A Locally Linear Procedure for Word Translation Soham Dan University of Pennsylvania sohamdan@seas.upenn.edu Hagai Taitelbaum Google Research hagait@google.com Jacob Goldberger Bar-Ilan University jacob.goldberger@biu.ac.il Abstract Learning a mapping between word embeddings of two languages given a dictionary is an important problem with several applications. A common mapping approach is using an orthogonal matrix. The Orthogonal Procrustes Analysis (PA) algorithm can be applied to find the optimal orthogonal matrix. This solution restricts the expressiveness of the translation model which may result in sub-optimal translations. We propose a natural extension of the PA algorithm that uses multiple orthogonal translation matrices to model the mapping and derive an algorithm to learn these multiple matrices. We achieve better performance in a bilingual word translation task and a cross lingual word similarity task compared to the single matrix baseline. We also show how multiple matrices can model multiple senses of a word. 1 Introduction Continuous word embeddings are a standard component in many NLP applications. The embedding spaces can exhibit similar structures across languages. Several studies (Mikolov et al., 2013; Klementiev et al., 2012) have exploited this similarity by learning a linear mapping from a source to a target embedding space, and demonstrated this approach on word translation tasks. Xing et al. (2015) showed that using orthogonal matrices can significantly improve performance. Since then, several studies have aimed at improving these bilingual word embeddings by using bilingual word dictionaries generated in either supervised or unsupervised manner. A multi-sense word in the source language can be translated into several different words in the target language. Studies have shown that for the multi-lingual translation problem, enforcing the transformation to be strictly orthogonal is too restrictive and performance can be improved by relaxing this constraint. It was shown that using orthogonalization as regularization rather than a strict constraint (Chen and Cardie, 2018), yields matrices that are close to orthogonal but are not necessarily exactly orthogonal, and improves performance (Taitelbaum et al., 2019a; Taitelbaum et al., 2019b). A single matrix however, whether orthogonal or not, cannot model the case of translating a multi-sense word. In this study, motivated by the multi-sense word situation, we investigated another way to relax the common modeling assumption that a single orthogonal transformation is suitable for translating all the words of a source language into a target language. In our approach, the word translation procedure is modeled by a set of orthogonal matrices where each word is translated by one of the matrices. The words are grouped according to their associated matrix. This grouping is determined by applying a clustering procedure on the vocabulary words. Next, for each cluster we learn a separate orthogonal matrix. The word mapping function is thus modeled by a local orthogonal transformation. When translating a new word, we select the most suitable matrix for this specific translation task. We illustrate our approach on several standard word translation tasks and we show performance …

Content truncated to 500 words