Combining Discrete and Continuous Features for Deterministic Transition-based Dependency Parsing

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1316–1321, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Combining Discrete and Continuous Features for Deterministic Transition-based Dependency Parsing Meishan Zhang and Yue Zhang Singapore University of Technology and Design {meishan zhang, yue zhang}@sutd.edu.sg Abstract We investigate a combination of a tra- ditional linear sparse feature model and a multi-layer neural network model for deterministic transition-based dependency parsing, by integrating the sparse features into the neural model. Correlations are drawn between the hybrid model and pre- vious work on integrating word embed- ding features into a discrete linear model. By analyzing the results of various parsers on web-domain parsing, we show that the integrated model is a better way to com- bine traditional and embedding features compared with previous methods. 1 Introduction Transition-based parsing algorithms construct out- put syntax trees using a sequence of shift-reduce actions. They are attractive in computational ef- ficiency, allowing linear time decoding with de- terministic (Nivre, 2008) or beam-search (Zhang and Clark, 2008) algorithms. Using rich non-local features, transition-based parsers achieve state-of- the-art accuracies for dependency parsing (Zhang and Nivre, 2011; Zhang and Nivre, 2012; Bohnet and Nivre, 2012; Choi and McCallum, 2013; Zhang et al., 2014). Deterministic transition-based parsers works by making a sequence of greedy local deci- sions (Nivre et al., 2004; Honnibal et al., 2013; Goldberg et al., 2014; G´omez-Rodr´ıguez and Fern´andez-Gonz´alez, 2015). They are attractive by very fast speeds. Traditionally, a linear model has been used for the local action classifier. Re- cently, Chen and Manning (2014) use a neural net- work (NN) to replace linear models, and report im- proved accuracies. A contrast between a neural network model and a linear model is shown in Figure 1 (a) and (b). · · · · · · (a) discrete linear (b) continuous NN · · · · · · (eg. MaltParser) (eg. Chen and Manning (2014)) (c) Turian et al. (2010) · · · · · · · · · transform (d) Guo et al. (2014) · · · · · · (e) this paper Figure 1: Five deterministic transition-based parsers with discrete and continuous features. A neural network model takes continuous vector representations of words as inputs, which can be pre-trained using large amounts of unlabeled data, thus containing more information. In addition, us- ing an extra hidden layer, a neural network is ca- pable of learning non-linear relations between au- tomatic features, achieving feature combinations automatically. Discrete manual features and continuous fea- tures complement each other. A natural question that arises from the contrast is whether traditional discrete features and continuous neural features can be integrated for better accuracies. We study this problem by constructing the neural network shown in Figure 1 (e), which incorporates the dis- crete input layer of the linear model (Figure 1 (a)) into the NN model (Figure 1 (b)) by conjoining 1316 it with the hidden layer. This architecture is con- nected with previous work on incorporating word embeddings …

Content truncated to 500 words