Unsupervised Joint Training of Bilingual Word Embeddings

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3224–3230 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3224 Unsupervised Joint Training of Bilingual Word Embeddings Benjamin Marie Atsushi Fujita National Institute of Information and Communications Technology 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan {bmarie, atsushi.fujita}@nict.go.jp Abstract State-of-the-art methods for unsupervised bilingual word embeddings (BWE) train a mapping function that maps pre-trained mono- lingual word embeddings into a bilingual space. Despite its remarkable results, unsuper- vised mapping is also well-known to be lim- ited by the dissimilarity between the original word embedding spaces to be mapped. In this work, we propose a new approach that trains unsupervised BWE jointly on synthetic paral- lel data generated through unsupervised ma- chine translation. We demonstrate that ex- isting algorithms that jointly train BWE are very robust to noisy training data and show that unsupervised BWE jointly trained signifi- cantly outperform unsupervised mapped BWE in several cross-lingual NLP tasks. 1 Introduction Bilingual word embeddings (BWE) represent the vocabulary of two languages in one common con- tinuous vector space. They are known to be useful in a wide range of cross-lingual NLP tasks. The most prevalent methods for training BWE are so-called mapping methods (Mikolov et al., 2013a): word embeddings for two languages are separately trained on respective monolingual data and then mapped into one common embedding space. The mapping function is usually trained us- ing a small bilingual lexicon for supervision. Re- cently, unsupervised mapping for BWE (Artetxe et al., 2018a; Lample et al., 2018a), i.e., trained without using any manually created bilingual re- sources, has been shown to reach a performance comparable to supervised BWE in several cross- lingual NLP tasks. Unsupervised BWE are trained with a three-step approach. First, word embed- dings are roughly mapped into an initial BWE space, for instance using adversarial training or an heuristic mapping. Then, using the initial BWE, a small synthetic bilingual lexicon is induced. Fi- nally, a new BWE, which is expected to be better than the initial BWE, is learned from the induced lexicon through a pseudo-supervision with some supervised mapping method. The last two steps can be repeated to refine the BWE. In spite of their success, unsupervised mapping methods are inherently limited by the dissimilar- ity between the original word embedding spaces to be mapped. The feasibility of aligning two em- bedding spaces relies on the assumption that they are isomorphic. However, Søgaard et al. (2018) showed that these spaces are, in general, far from being isomorphic, and thus they result in sub- optimal or degenerated unsupervised mappings. On the other hand, supervised methods that jointly train BWE from scratch (Upadhyay et al., 2016), on parallel or comparable corpora, do not have such limits since no pre-existing embedding spaces and no mapping function are involved. These methods jointly train BWE by exploiting bilingual and monolingual contexts of words, ma- terialized by sentence or document pairs, to learn a single BWE space. However, they …

Content truncated to 500 words