Delexicalized Word Embeddings for Cross-lingual Dependency Parsing

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 241–250, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Delexicalized Word Embeddings for Cross-lingual Dependency Parsing Mathieu Dehouck1 and Pascal Denis2 1Univ. Lille, CNRS, UMR 9189 - CRIStAL 1,2Magnet Team, Inria Lille - Nord Europe 59650 Villeneuve d’Ascq, France {mathieu.dehouck, pascal.denis}@inria.fr Abstract This paper presents a new approach to the problem of cross-lingual dependency parsing, aiming at leveraging training data from different source languages to learn a parser in a target language. Specifi- cally, this approach first constructs word vector representations that exploit struc- tural (i.e., dependency-based) contexts but only considering the morpho-syntactic in- formation associated with each word and its contexts. These delexicalized word em- beddings, which can be trained on any set of languages and capture features shared across languages, are then used in com- bination with standard language-specific features to train a lexicalized parser in the target language. We evaluate our approach through experiments on a set of eight dif- ferent languages that are part the Univer- sal Dependencies Project. Our main re- sults show that using such delexicalized embeddings, either trained in a monolin- gual or multilingual fashion, achieves sig- nificant improvements over monolingual baselines. 1 Introduction Over the recent years, distributional and dis- tributed representations of words have become a critical component of many NLP systems (Turian et al., 2010; Collobert et al., 2011). The reason for this success is that these low-dimensional, dense word vectors address two major problems that ap- pear in many NLP applications, namely data spar- sity and the inherent lack of expressivity of one- hot representations, and they can also be trained on large unannotated corpora which are cheap to pro- duce and easily available. While they have proven useful in a number of tasks, and especially in de- pendency parsing (Koo et al., 2008), these word vectors are often learned in a generic manner, only using linear bag-of-word contexts (F. Brown et al., 1992; Mikolov et al., 2013), without pay- ing much attention to the specifics of the task to be solved. Only very recently, people have tried to learn dependency-based embeddings (Bansal et al., 2014; Levy and Goldberg, 2014; Mad- hyastha et al., 2014; Bansal, 2015), and these new structure-aware representations have been shown to improve parsing performance in a monolingual setting. We would like to generalize this idea to a multi- lingual setting in a way that allows the transfer of structural information associated with words from one (or several) languages to others. While pre- vious work have attempted to learn multilingual word clusters or embeddings (Guo et al., 2015) and use these for cross-lingual transfer, this pa- per explores a different research direction. Specif- ically, we investigate the use of vectorial rep- resentations in which lemma information have been abstracted away from both words and con- texts, hence reduced to their morpho-syntactic at- tributes. The appeal of these de facto delexicalized word representations is …

Content truncated to 500 words