Adversarial Training for Unsupervised Bilingual Lexicon Induction

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1959–1970 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1179 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1959–1970 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1179 Adversarial Training for Unsupervised Bilingual Lexicon Induction Meng Zhang†‡ Yang Liu†‡∗Huanbo Luan† Maosong Sun†‡ †State Key Laboratory of Intelligent Technology and Systems Tsinghua National Laboratory for Information Science and Technology Department of Computer Science and Technology, Tsinghua University, Beijing, China ‡Jiangsu Collaborative Innovation Center for Language Competence, Jiangsu, China zmlarry@foxmail.com, liuyang2011@tsinghua.edu.cn luanhuanbo@gmail.com, sms@tsinghua.edu.cn Abstract Word embeddings are well known to cap- ture linguistic regularities of the language on which they are trained. Researchers also observe that these regularities can transfer across languages. However, previ- ous endeavors to connect separate mono- lingual word embeddings typically require cross-lingual signals as supervision, either in the form of parallel corpus or seed lex- icon. In this work, we show that such cross-lingual connection can actually be established without any form of supervi- sion. We achieve this end by formulating the problem as a natural adversarial game, and investigating techniques that are cru- cial to successful training. We carry out evaluation on the unsupervised bilingual lexicon induction task. Even though this task appears intrinsically cross-lingual, we are able to demonstrate encouraging per- formance without any cross-lingual clues. 1 Introduction As word is the basic unit of a language, the better- ment of its representation has significant impact on various natural language processing tasks. Con- tinuous word representations, commonly known as word embeddings, have formed the basis for numerous neural network models since their ad- vent. Their popularity results from the perfor- mance boost they bring, which should in turn be attributed to the linguistic regularities they capture (Mikolov et al., 2013b). Soon following the success on monolingual tasks, the potential of word embeddings for cross- lingual natural language processing has attracted much attention. In their pioneering work, Mikolov ∗Corresponding author. caballo (horse) cerdo (pig) gato (cat) horse pig cat Spanish English Figure 1: Illustrative monolingual word em- beddings of Spanish and English, adapted from (Mikolov et al., 2013a). Although trained inde- pendently, the two sets of embeddings exhibit ap- proximate isomorphism. et al. (2013a) observe that word embeddings trained separately on monolingual corpora exhibit isomorphic structure across languages, as illus- trated in Figure 1. This interesting finding is in line with research on human cognition (Youn et al., 2016). It also means a linear transforma- tion may be established to connect word embed- ding spaces, allowing word feature transfer. This has far-reaching implication on low-resource sce- narios (Daum´e III and Jagarlamudi, 2011; Irvine and Callison-Burch, 2013), because word embed- dings only require plain text to train, which is the most abundant form of linguistic resource. However, connecting separate word embedding spaces typically requires supervision from cross- lingual signals. For example, Mikolov et al. (2013a) use five thousand seed word …

Content truncated to 500 words