Multilingual Training of Crosslingual Word Embeddings

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 894–904, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Multilingual Training of Crosslingual Word Embeddings Long Duong,1 Hiroshi Kanayama,2 Tengfei Ma,3 Steven Bird14 and Trevor Cohn1 1Department of Computing and Information Systems, University of Melbourne 2IBM Research – Tokyo 3IBM T.J. Watson Research Center 4International Computer Science Institute, University of California Berkeley Abstract Crosslingual word embeddings represent lexical items from different languages using the same vector space, enabling crosslingual transfer. Most prior work constructs embeddings for a pair of lan- guages, with English on one side. We in- vestigate methods for building high qual- ity crosslingual word embeddings for many languages in a unified vector space. In this way, we can exploit and combine information from many languages. We re- port competitive performance on bilingual lexicon induction, monolingual similarity and crosslingual document classification tasks. 1 Introduction Monolingual word embeddings have facilitated advances in many natural language process- ing tasks, such as natural language understand- ing (Collobert and Weston, 2008), sentiment anal- ysis (Socher et al., 2013), and dependency pars- ing (Dyer et al., 2015). Crosslingual word embed- dings represent words from several languages in the same low dimensional space. They are help- ful for multilingual tasks such as machine trans- lation (Brown et al., 1993) and bilingual named entity recognition (Wang et al., 2013). Crosslin- gual word embeddings can also be used in trans- fer learning, where the source model is trained on one language and applied directly to another lan- guage; this is suitable for the low-resource sce- nario (Yarowsky and Ngai, 2001; Duong et al., 2015b; Das and Petrov, 2011; T¨ackstr¨om et al., 2012). Most prior work on building crosslingual word embeddings focuses on a pair of languages. En- glish is usually on one side, thanks to the wealth of available English resources. However, it is highly desirable to have a crosslingual word em- beddings for many languages so that different rela- tions can be exploited.1 For example, since Italian and Spanish are similar, they are excellent candi- dates for transfer learning. However, few parallel resources exist between Italian and Spanish for di- rectly building bilingual word embeddings. Our multilingual word embeddings, on the other hand, map both Italian and Spanish to the same space without using any direct bilingual signal between them. In addition, multilingual word embeddings allow multiple source language transfer learning, producing a more general model and overcoming data sparseness (McDonald et al., 2011; Guo et al., 2016; Agi´c et al., 2016). Moreover, multilin- gual word embeddings are also crucial for multi- lingual applications such as multi-source machine translation (Zoph and Knight, 2016), and multi- source transfer dependency parsing (McDonald et al., 2011; Duong et al., 2015a). We propose several algorithms to map bilingual word embeddings to the same vector space, ei- ther during training or during post-processing. We apply a linear transformation to map the English side of each pretrained crosslingual word …

Content truncated to 500 words