Weakly-Supervised Concept-based Adversarial Learning for Cross-lingual Word Embeddings

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4419–4430, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4419 Weakly-Supervised Concept-based Adversarial Learning for Cross-lingual Word Embeddings Haozhou Wang1, James Henderson2, Paola Merlo1 1 University of Geneva 2 Idiap Research Institute {haozhou.wang, paola.merlo}@unige.ch, james.henderson@idiap.ch Abstract Distributed representations of words which map each word to a continuous vector have proven useful in capturing important linguis- tic information not only in a single language but also across different languages. Current unsupervised adversarial approaches show that it is possible to build a mapping matrix that aligns two sets of monolingual word embed- dings without high quality parallel data, such as a dictionary or a sentence-aligned corpus. However, without an additional step of refine- ment, the preliminary mapping learnt by these methods is unsatisfactory, leading to poor per- formance for typologically distant languages. In this paper, we propose a weakly-supervised adversarial training method to overcome this limitation, based on the intuition that map- ping across languages is better done at the concept level than at the word level. We propose a concept-based adversarial training method which improves the performance of previous unsupervised adversarial methods for most languages, and especially for typologi- cally distant language pairs. 1 Introduction Distributed representations of words which map each word to a continuous vector have proven use- ful in capturing important linguistic information. Vectors of words that are semantically or syntacti- cally similar have been shown to be close to each other in the same space (Mikolov et al., 2013a,c; Pennington et al., 2014), making them widely use- ful in many natural language processing tasks such as machine translation and parsing (Bansal et al., 2014; Mi et al., 2016), both in a single language and across different languages. Mikolov et al. (2013b) first observed that the ge- ometric positions of similar words in different lan- guages were related by a linear relation. Zou et al. (2013) showed that a cross-lingually shared word embedding space is more useful than a monolin- gual space in an end-to-end machine translation task. However, traditional methods for mapping two monolingual word embeddings require high quality aligned sentences or dictionaries (Faruqui and Dyer, 2014; Ammar et al., 2016b). Reducing the need for parallel data, then, has become the main issue for cross-lingual word em- bedding mapping. Some recent work aiming at reducing resources has shown competitive cross- lingual mappings across similar languages, us- ing a pseudo-dictionary, such as identical charac- ter strings between two languages (Smith et al., 2017), or a simple list of numerals (Artetxe et al., 2017). In a more general method, Zhang et al. (2017) have shown that learning mappings across lan- guages via adversarial training (Goodfellow et al., 2014) can avoid using bilingual evidence. This generality comes at the expense of performance. To overcome this limitation, Lample et al. (2018) refine the preliminary mapping matrix trained by generative adversarial networks (GANs) and ob- tain …

Content truncated to 500 words