Cross-lingual Models of Word Embeddings: An Empirical Comparison

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1661–1670, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Cross-lingual Models of Word Embeddings: An Empirical Comparison Shyam Upadhyay1 Manaal Faruqui2 Chris Dyer2 Dan Roth1 1 Department of Computer Science, University of Illinois, Urbana-Champaign, IL, USA 2 School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA upadhya3@illinois.edu, mfaruqui@cs.cmu.edu cdyer@cs.cmu.edu, danr@illinois.edu Abstract Despite interest in using cross-lingual knowledge to learn word embeddings for various tasks, a systematic comparison of the possible approaches is lacking in the literature. We perform an extensive eval- uation of four popular approaches of in- ducing cross-lingual embeddings, each re- quiring a different form of supervision, on four typologically different language pairs. Our evaluation setup spans four dif- ferent tasks, including intrinsic evaluation on mono-lingual and cross-lingual simi- larity, and extrinsic evaluation on down- stream semantic and syntactic applica- tions. We show that models which require expensive cross-lingual knowledge almost always perform better, but cheaply super- vised models often prove competitive on certain tasks. 1 Introduction Learning word vector representations using mono- lingual distributional information is now a ubiqui- tous technique in NLP. The quality of these word vectors can be significantly improved by incor- porating cross-lingual distributional information (Klementiev et al., 2012; Zou et al., 2013; Vuli´c and Moens, 2013b; Mikolov et al., 2013b; Faruqui and Dyer, 2014; Hermann and Blunsom, 2014; Chandar et al., 2014, inter alia), with improve- ments observed both on monolingual (Faruqui and Dyer, 2014; Rastogi et al., 2015) and cross-lingual tasks (Guo et al., 2015; Søgaard et al., 2015; Guo et al., 2016). Several models for inducing cross-lingual em- beddings have been proposed, each requiring a dif- ferent form of cross-lingual supervision – some can use document-level alignments (Vuli´c and Moens, 2015), others need alignments at the sen- tence (Hermann and Blunsom, 2014; Gouws et al., 2015) or word level (Faruqui and Dyer, 2014; Gouws and Søgaard, 2015), while some require both sentence and word alignments (Luong et al., 2015). However, a systematic comparison of these models is missing from the literature, making it difficult to analyze which approach is suitable for a particular NLP task. In this paper, we fill this void by empirically comparing four cross-lingual word embedding models each of which require different form of alignment(s) as supervision, across several dimensions. To this end, we train these models on four different language pairs, and evaluate them on both monolingual and cross-lingual tasks.1 First, we show that different models can be viewed as instances of a more general frame- work for inducing cross-lingual word embeddings. Then, we evaluate these models on both extrin- sic and intrinsic tasks. Our intrinsic evaluation assesses the quality of the vectors on monolin- gual (§4.2) and cross-lingual (§4.3) word simi- larity tasks, while our extrinsic evaluation spans semantic (cross-lingual document classification §4.4) and syntactic tasks (cross-lingual depen- dency parsing §4.5). Our experiments show that word vectors trained using expensive cross-lingual supervision (word alignments or sentence alignments) perform the …

Content truncated to 500 words