Unbounded Dependency Recovery for Parser Evaluation

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 813–821, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Unbounded Dependency Recovery for Parser Evaluation Laura Rimell and Stephen Clark University of Cambridge Computer Laboratory laura.rimell@cl.cam.ac.uk stephen.clark@cl.cam.ac.uk Mark Steedman University of Edinburgh School of Informatics steedman@inf.ed.ac.uk Abstract This paper introduces a new parser eval- uation corpus containing around 700 sen- tences annotated with unbounded depen- dencies, from seven different grammatical constructions. We run a series of off-the- shelf parsers on the corpus to evaluate how well state-of-the-art parsing technology is able to recover such dependencies. The overall results range from 25% accuracy to 59%. These low scores call into ques- tion the validity of using Parseval scores as a general measure of parsing capability. We discuss the importance of parsers be- ing able to recover unbounded dependen- cies, given their relatively low frequency in corpora. We also analyse the various er- rors made on these constructions by one of the more successful parsers. 1 Introduction Statistical parsers are now obtaining Parseval scores of over 90% on the WSJ section of the Penn Treebank (Bod, 2003; Petrov and Klein, 2007; Huang, 2008; Carreras et al., 2008). McClosky et al. (2006) report an F-score of 92.1% using self- training applied to the reranker of Charniak and Johnson (2005). Such scores, in isolation, may suggest that statistical parsing is close to becom- ing a solved problem, and that further incremental improvements will lead to parsers becoming as ac- curate as POS taggers. A single score in isolation can be misleading, however, for a number of reasons. First, the single score is an aggregate over a highly skewed distri- bution of all constituent types; evaluations which look at individual constituent or dependency types show that the accuracies on some, semantically important, constructions, such as coordination and PP-attachment, are much lower (Collins, 1999). Second, it is well known that the accuracy of parsers trained on the Penn Treebank degrades when they are applied to different genres and do- mains (Gildea, 2001). Finally, some researchers have argued that the Parseval metrics (Black et al., 1991) are too forgiving with respect to certain er- rors and that an evaluation based on syntactic de- pendencies, for which scores are typically lower, is a better test of parser performance (Lin, 1995; Carroll et al., 1998). In this paper we focus on the first issue, that the performance of parsers on some constructions is much lower than the overall score. The construc- tions that we focus on are various unbounded de- pendency constructions. These are interesting for parser evaluation for the following reasons: one, they provide a strong test of the parser’s knowl- edge of the grammar of the language, since many instances of unbounded dependencies are diffi- cult to recover using shallow techniques in which the grammar is only superficially represented; and two, recovering these dependencies is necessary to completely represent the underlying predicate- argument structure of a sentence, useful for appli- cations such as Question …

Content truncated to 500 words