Cross-lingual Word Clusters for Direct Transfer of Linguistic Structure

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 477–487, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Cross-lingual Word Clusters for Direct Transfer of Linguistic Structure Oscar T¨ackstr¨om∗ SICS / Uppsala University Kista / Uppsala, Sweden oscar@sics.se Ryan McDonald Google New York, NY ryanmcd@google.com Jakob Uszkoreit Google Mountain View, CA uszkoreit@google.com Abstract It has been established that incorporating word cluster features derived from large unlabeled corpora can significantly improve prediction of linguistic structure. While previous work has focused primarily on English, we extend these results to other languages along two dimen- sions. First, we show that these results hold true for a number of languages across families. Second, and more interestingly, we provide an algorithm for inducing cross-lingual clusters and we show that features derived from these clusters significantly improve the accuracy of cross-lingual structure prediction. Specifically, we show that by augmenting direct-transfer sys- tems with cross-lingual cluster features, the rel- ative error of delexicalized dependency parsers, trained on English treebanks and transferred to foreign languages, can be reduced by up to 13%. When applying the same method to di- rect transfer of named-entity recognizers, we observe relative improvements of up to 26%. 1 Introduction The ability to predict the linguistic structure of sen- tences or documents is central to the field of nat- ural language processing (NLP). Structures such as named-entity tag sequences (Bikel et al., 1999) or sen- timent relations (Pang and Lee, 2008) are inherently useful in data mining, information retrieval and other user-facing technologies. More fundamental struc- tures such as part-of-speech tag sequences (Ratna- parkhi, 1996) or syntactic parse trees (Collins, 1997; K¨ubler et al., 2009), on the other hand, comprise the core linguistic analysis for many important down- stream tasks such as machine translation (Chiang, ∗The majority of this work was performed while the author was an intern at Google, New York, NY. 2005; Collins et al., 2005). Currently, supervised data-driven methods dominate the literature on lin- guistic structure prediction (Smith, 2011). Regret- tably, the majority of studies on these methods have focused on evaluations specific to English, since it is the language with the most annotated resources. No- table exceptions include the CoNLL shared tasks (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003; Buchholz and Marsi, 2006; Nivre et al., 2007) and subsequent studies on this data, as well as a number of focused studies on one or two specific languages, as discussed by Bender (2011). While annotated resources for parsing and several other tasks are available in a number of languages, we cannot expect to have access to labeled resources for all tasks in all languages. This fact has given rise to a large body of research on unsupervised (Klein and Manning, 2004), semi-supervised (Koo et al., 2008) and transfer (Hwa et al., 2005) systems for prediction of linguistic structure. These methods all attempt to benefit from the plethora of unlabeled monolingual and/or cross-lingual data that has become available …

Content truncated to 500 words