Evaluation of Dependency Parsers on Unbounded Dependencies

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 833–841, Beijing, August 2010 Evaluation of Dependency Parsers on Unbounded Dependencies Joakim Nivre Laura Rimell Ryan McDonald Carlos G´omez-Rodr´ıguez Uppsala University Univ. of Cambridge Google Inc. Universidade da Coru˜na joakim.nivre@lingfil.uu.se laura.rimell@cl.cam.ac.uk ryanmcd@google.com cgomezr@udc.es Abstract We evaluate two dependency parsers, MSTParser and MaltParser, with respect to their capacity to recover unbounded de- pendencies in English, a type of evalu- ation that has been applied to grammar- based parsers and statistical phrase struc- ture parsers but not to dependency parsers. The evaluation shows that when combined with simple post-processing heuristics, the parsers correctly recall unbounded dependencies roughly 50% of the time, which is only slightly worse than two grammar-based parsers specifically de- signed to cope with such dependencies. 1 Introduction Though syntactic parsers for English are re- ported to have accuracies over 90% on the Wall Street Journal (WSJ) section of the Penn Tree- bank (PTB) (McDonald et al., 2005; Sagae and Lavie, 2006; Huang, 2008; Carreras et al., 2008), broad-coverage parsing is still far from being a solved problem. In particular, metrics like attach- ment score for dependency parsers (Buchholz and Marsi, 2006) and Parseval for constituency parsers (Black et al., 1991) suffer from being an aver- age over a highly skewed distribution of differ- ent grammatical constructions. As a result, in- frequent yet semantically important construction types could be parsed with accuracies far below what one might expect. This shortcoming of aggregate parsing met- rics was highlighted in a recent study by Rimell et al. (2009), introducing a new parser evalua- tion corpus containing around 700 sentences an- notated with unbounded dependencies in seven different grammatical constructions. This corpus was used to evaluate five state-of-the-art parsers for English, focusing on grammar-based and sta- tistical phrase structure parsers. For example, in the sentence By Monday, they hope to have a sheaf of documents both sides can trust., parsers should recognize that there is a dependency be- tween trust and documents, an instance of object extraction out of a (reduced) relative clause. In the evaluation, the recall of state-of-the-art parsers on this kind of dependency varies from a high of 65% to a low of 1%. When averaging over the seven constructions in the corpus, none of the parsers had an accuracy higher than 61%. In this paper, we extend the evaluation of Rimell et al. (2009) to two dependency parsers, MSTParser (McDonald, 2006) and MaltParser (Nivre et al., 2006a), trained on data from the PTB, converted to Stanford typed dependencies (de Marneffe et al., 2006), and combined with a simple post-processor to extract unbounded de- pendencies from the basic dependency tree. Ex- tending the evaluation to dependency parsers is of interest because it sheds light on whether highly tuned grammars or computationally expensive parsing formalisms are necessary for extracting complex linguistic phenomena in practice. Unlike the best performing grammar-based parsers stud- ied in Rimell et al. (2009), neither MSTParser nor MaltParser was developed specifically as a parser for English, …

Content truncated to 500 words