Feature Embedding for Dependency Parsing

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 816–826, Dublin, Ireland, August 23-29 2014. Feature Embedding for Dependency Parsing Wenliang Chen†, Yue Zhang‡, and Min Zhang†∗ †School of Computer Science and Technology, Soochow University, China ‡Singapore University of Technology and Design, Singapore {wlchen, mzhang}@suda.edu.cn yue zhang@sutd.edu.sg Abstract In this paper, we propose an approach to automatically learning feature embeddings to address the feature sparseness problem for dependency parsing. Inspired by word embeddings, feature embeddings are distributed representations of features that are learned from large amounts of auto-parsed data. Our target is to learn feature embeddings that can not only make full use of well-established hand-designed features but also benefit from the hidden-class representations of features. Based on feature embeddings, we present a set of new features for graph-based dependency parsing models. Experiments on the standard Chinese and English data sets show that the new parser achieves significant performance improvements over a strong baseline. 1 Introduction Discriminative models have become the dominant approach for dependency parsing (Nivre et al., 2007; Zhang and Clark, 2008; Hatori et al., 2011). State-of-the-art accuracies have been achieved by the use of rich features in discriminative models (Carreras, 2007; Koo and Collins, 2010; Bohnet, 2010; Zhang and Nivre, 2011). While lexicalized features extracted from non-local contexts enhance the discriminative power of parsers, they are relatively sparse. Given a limited set of training data (typically less than 50k sentences for dependency parsing), the chance of a feature occurring in the training data but not in the test data can be high. Another limitation on features is that many are typically derived by (manual) combination of atomic features. For example, given the head word (wh) and part-of-speech tag (ph), dependent word (wd) and part-of-speech tag (pd), and the label (l) of a dependency arc, state-of-the-art dependency parsers can have the combined features: [wh; ph], [wh; ph; wd; pd], [wh; ph; wd], and so on, in addition to the atomic features: [wh], [ph], etc. Such combination is necessary for high accuracies because the dominant approach uses linear models, which can not capture complex correlations between atomic features. We tackle the above issues by borrowing solutions from word representations, which have been in- tensely studied in the NLP community (Turian et al., 2010). In particular, distributed representations of words have been used for many NLP problems, which represent a word by information from the words it frequently co-occurs with (Lin, 1997; Curran, 2005; Collobert et al., 2011; Bengio, 2009; Mikolov et al., 2013b). The representation can be learned from large amounts of raw sentences, and hence used to reduce OOV rates in test data. In addition, since the representation of each word carries information about its context words, it can also be used to calculate word similarity (Mikolov et al., 2013a), or used as additional semantic features (Koo et al., 2008). In this paper, we show that a distributed representation can be learned for features also. Learned from large amount of automatically parsed data, …

Content truncated to 500 words