Earth Mover’s Distance Minimization for Unsupervised Bilingual Lexicon Induction

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1934–1945 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Earth Mover’s Distance Minimization for Unsupervised Bilingual Lexicon Induction Meng Zhang†‡ Yang Liu†‡ Huanbo Luan† Maosong Sun†‡ †State Key Laboratory of Intelligent Technology and Systems Tsinghua National Laboratory for Information Science and Technology Department of Computer Science and Technology, Tsinghua University, Beijing, China ‡Jiangsu Collaborative Innovation Center for Language Competence, Jiangsu, China zmlarry@foxmail.com, liuyang2011@tsinghua.edu.cn luanhuanbo@gmail.com, sms@tsinghua.edu.cn Abstract Cross-lingual natural language processing hinges on the premise that there exists in- variance across languages. At the word level, researchers have identified such in- variance in the word embedding seman- tic spaces of different languages. How- ever, in order to connect the separate spaces, cross-lingual supervision encoded in parallel data is typically required. In this paper, we attempt to establish the cross-lingual connection without relying on any cross-lingual supervision. By viewing word embedding spaces as dis- tributions, we propose to minimize their earth mover’s distance, a measure of diver- gence between distributions. We demon- strate the success on the unsupervised bilingual lexicon induction task. In addi- tion, we reveal an interesting finding that the earth mover’s distance shows potential as a measure of language difference. 1 Introduction Despite tremendous variation and diversity, lan- guages are believed to share something in com- mon. Indeed, this belief forms the underlying ba- sis of computational approaches to cross-lingual transfer (T¨ackstr¨om et al., 2013, inter alia), other- wise it would be inconceivable for the transfer to successfully generalize. Linguistic universals manifest themselves at various levels of linguistic units. At the word level, there is evidence that different languages represent concepts with similar structure (Youn et al., 2016). Interestingly, as computational mod- els of word semantics, monolingual word embed- dings also exhibit isomorphism across languages (Mikolov et al., 2013a). This finding opens up the possibility to use a simple transformation, e.g. a linear map, to connect separately trained word em- beddings cross-lingually. Learning such a trans- formation typically calls for cross-lingual supervi- sion from parallel data (Faruqui and Dyer, 2014; Lu et al., 2015; Dinu et al., 2015; Lazaridou et al., 2015; Smith et al., 2017). In this paper, we ask the question: Can we un- cover the transformation without any cross-lingual supervision? At first sight, this task appears formidable, as it would imply that a bilingual se- mantic space can be constructed by using mono- lingual corpora only. On the other hand, the ex- istence of structural isomorphism across mono- lingual embedding spaces points to the feasibility of this task: The transformation exists right there only to be discovered by the right tool. We propose such a tool to answer the above question in the affirmative. The key insight is to view embedding spaces as distributions, and the desired transformation should make the two dis- tributions close. This naturally calls for a measure of distribution closeness, for which we introduce the earth mover’s distance. Therefore, our task can be formulated as the minimization …

Content truncated to 500 words