Parser Showdown at the Wall Street Corral: An Empirical Investigation of Error Types in Parser Output

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1048–1059, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Parser Showdown at the Wall Street Corral: An Empirical Investigation of Error Types in Parser Output Jonathan K. Kummerfeld† David Hall† James R. Curran‡ Dan Klein† †Computer Science Division ‡ e -lab, School of IT University of California, Berkeley University of Sydney Berkeley, CA 94720, USA Sydney, NSW 2006, Australia {jkk,dlwh,klein}@cs.berkeley.edu james@it.usyd.edu.au Abstract Constituency parser performance is primarily interpreted through a single metric, F-score on WSJ section 23, that conveys no linguis- tic information regarding the remaining errors. We classify errors within a set of linguisti- cally meaningful types using tree transforma- tions that repair groups of errors together. We use this analysis to answer a range of ques- tions about parser behaviour, including what linguistic constructions are difficult for state- of-the-art parsers, what types of errors are be- ing resolved by rerankers, and what types are introduced when parsing out-of-domain text. 1 Introduction Parsing has been a major area of research within computational linguistics for decades, and con- stituent parser F-scores on WSJ section 23 have ex- ceeded 90% (Petrov and Klein, 2007), and 92% when using self-training and reranking (McClosky et al., 2006; Charniak and Johnson, 2005). While these results give a useful measure of overall per- formance, they provide no information about the na- ture, or relative importance, of the remaining errors. Broad investigations of parser errors beyond the PARSEVAL metric (Abney et al., 1991) have either focused on specific parsers, e.g. Collins (2003), or have involved conversion to dependencies (Carroll et al., 1998; King et al., 2003). In all of these cases, the analysis has not taken into consideration how a set of errors can have a common cause, e.g. a single mis-attachment can create multiple node errors. We propose a new method of error classifica- tion using tree transformations. Errors in the parse tree are repaired using subtree movement, node cre- ation, and node deletion. Each step in the process is then associated with a linguistically meaningful er- ror type, based on factors such as the node that is moved, its siblings, and parents. Using our method we analyse the output of thir- teen constituency parsers on newswire. Some of the frequent error types that we identify are widely recognised as challenging, such as prepositional phrase (PP) attachment. However, other significant types have not received as much attention, such as clause attachment and modifier attachment. Our method also enables us to investigate where reranking and self-training improve parsing. Pre- viously, these developments were analysed only in terms of their impact on F-score. Similarly, the chal- lenge of out-of-domain parsing has only been ex- pressed in terms of this single objective. We are able to decompose the drop in performance and show that a disproportionate number of the extra errors are due to coordination and clause attachment. This work presents a comprehensive investigation of parser …

Content truncated to 500 words