Multilingual Factor Analysis

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1738–1750 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1738 Multilingual Factor Analysis Francisco Vargas, Kamen Brestnichki, Alex Papadopoulos-Korfiatis and Nils Hammerla Babylon Health {firstname.lastname, alex.papadopoulos}@babylonhealth.com Abstract In this work we approach the task of learn- ing multilingual word representations in an of- fline manner by fitting a generative latent vari- able model to a multilingual dictionary. We model equivalent words in different languages as different views of the same word gener- ated by a common latent variable representing their latent lexical meaning. We explore the task of alignment by querying the fitted model for multilingual embeddings achieving com- petitive results across a variety of tasks. The proposed model is robust to noise in the em- bedding space making it a suitable method for distributed representations learned from noisy corpora. 1 Introduction Popular approaches for multilingual alignment of word embeddings base themselves on the obser- vation in (Mikolov et al., 2013a), which noticed that continuous word embedding spaces (Mikolov et al., 2013b; Pennington et al., 2014; Bojanowski et al., 2017; Joulin et al., 2017) exhibit similar structures across languages. This observation has led to multiple successful methods in which a di- rect linear mapping between the two spaces is learned through a least squares based objective (Mikolov et al., 2013a; Smith et al., 2017; Xing et al., 2015) using a paired bilingual dictionary. An alternate set of approaches based on Canon- ical Correlation Analysis (CCA) (Knapp, 1978) seek to project monolingual embeddings into a shared multilingual space (Faruqui and Dyer, 2014b; Lu et al., 2015). Both these methods aim to exploit the correlations between the monolingual vector spaces when projecting into the aligned multilingual space. The multilingual embeddings from (Faruqui and Dyer, 2014b; Lu et al., 2015) are shown to improve on word level semantic tasks, which sustains the authors’ claim that mul- tilingual information enhances semantic spaces. In this paper we present a new non-iterative method based on variants of factor analysis (Browne, 1979; McDonald, 1970; Browne, 1980) for aligning monolingual representations into a multilingual space. Our generative modelling as- sumes that a single word translation pair is gen- erated by an embedding representing the lexical meaning of the underlying concept. We achieve competitive results across a wide range of tasks compared to state-of-the-art methods, and we con- jecture that our multilingual latent variable model has sound generative properties that match those of psycholinguistic theories of the bilingual mind (Weinreich, 1953). Furthermore, we show how our model extends to more than two languages within the generative framework which is some- thing that previous alignment models are not nat- urally suited to, instead resorting to combining bilingual models with a pivot as in (Ammar et al., 2016). Additionally the general benefit of the proba- bilistic setup as discussed in (Tipping and Bishop, 1999) is that it offers the potential to extend the scope of conventional alignment methods to model and exploit linguistic …

Content truncated to 500 words