Neural Scoring Function for MST Parser

Neural Scoring Function for MST Parser Jindˇrich Libovick´y Charles University in Prague Malostransk´e n´amˇest´ı 25, Praha, Czech Republic libovicky@ufal.mff.cuni.cz Abstract Continuous word representations appeared to be a useful feature in many natural language processing tasks. Using fixed-dimension pre-trained word embeddings allows avoiding sparse bag-of-words representation and to train models with fewer parameters. In this paper, we use fixed pre-trained word embeddings as additional features for a neural scoring function in the MST parser. With the multi-layer architecture of the scoring function we can avoid handcrafting feature conjunctions. The continuous word representations on the input also allow us to reduce the number of lexical features, make the parser more robust to out-of-vocabulary words, and reduce the total number of parameters of the model. Although its accuracy stays below the state of the art, the model size is substantially smaller than with the standard features set. Moreover, it performs well for languages where only a smaller treebank is available and the results promise to be useful in cross-lingual parsing. Keywords: dependency parsing, MST Parser, word embeddings, Matrix-tree theorem 1. Introduction Syntactic parsing is one of the oldest problems studied by Computational Linguistics. Recently, dependency parsing gained big popularity with graph-based (McDonald et al., 2005) and transition-based parsing (Nivre, 2006) being two major paradigms. In this paper, we work with MST parser – the most of- ten used graph-based approach. We present several exper- iments showing how pre-trained word embeddings could be used instead of explicit categorial (one-hot) lexical fea- tures. Training of the word embeddings is based on lan- guage modeling, so it can be trained on plain text with any further annotation. They can be computed on much bigger data than the treebanks used for training the parser. By having the lexical information in form of embeddings, the model can learn how to deal with words that were not seen in the training part of the treebank (out-of-vocabulary words). The simple neural scoring function that we propose can be used to avoid handcrafting feature templates for feature conjunctions. We use the Matrix-Tree theorem (Chaiken and Kleitman, 1978) to sum all possible sentence parses to be able to use a gradient learning procedure as proposed by Koo et al. (2007). Using such a scoring function also allows to control the number of parameters of the model. In the next section, we summarize related work. Section 3. describes the model we use for training. In Section 4. we evaluate the model. Section 5. then concludes the work. 2. Related Work In the MSTParser by McDonald et al. (2005), parsing is modeled as finding the maximum spanning tree in a weighted oriented complete graph with vertices represent- ing words of the sentence. The purpose of the learning pro- cedure is therefore to find a function that favors the edges which are actual dependencies in the sentence. The advan- tage of using such a model is that it can naturally produce non-projective parses. Bansal et al. (2014) tried to use word embeddings to im- prove …

Content truncated to 500 words