Identifying Cascading Errors using Constraints in Dependency Parsing

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1148–1158, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Identifying Cascading Errors using Constraints in Dependency Parsing Dominick Ng and James R. Curran @-lab, School of Information Technologies University of Sydney NSW 2006, Australia {dominick.ng,james.r.curran}@sydney.edu.au Abstract Dependency parsers are usually evaluated on attachment accuracy. Whilst easily in- terpreted, the metric does not illustrate the cascading impact of errors, where the parser chooses an incorrect arc, and is sub- sequently forced to choose further incor- rect arcs elsewhere in the parse. We apply arc-level constraints to MST- parser and ZPar, enforcing the correct analysis of specific error classes, whilst otherwise continuing with decoding. We investigate the direct and indirect impact of applying constraints to the parser. Er- roneous NP and punctuation attachments cause the most cascading errors, while in- correct PP and coordination attachments are frequent but less influential. Punctu- ation is especially challenging, as it has long been ignored in parsing, and serves a variety of disparate syntactic roles. 1 Introduction Dependency parsers are evaluated using word- level attachment accuracy. Whilst comparable across systems, this does not provide insight into why the parser makes certain errors, or whether certain misattachments are caused by other errors. For example, incorrectly identifying a modifier head may only introduce a single attachment error, while misplacing the root of a sentence will create substantially more errors elsewhere. In projective dependency parsing, erroneous arcs can also force the parser to select other incorrect arcs. Kummerfeld et al. (2012) propose a static post- parsing analysis to categorise groups of bracket er- rors in constituency parsing into higher level error classes such as clause attachment. However, this cannot account for cascading changes resulting from repairing errors, or limitations which may prevent the parser from applying a repair. It is un- clear whether the parser will apply the repair op- eration in its entirety, or if it will introduce other changes in response to the repairs. We develop an evaluation procedure to evalu- ate the influence of each error class in dependency parsing without making assumptions about how the parser will behave. We define error classes based on dependency labels, and use the depen- dencies in each class as arc constraints specifying the correct head and label for particular words in each sentence. We adapt parsers to apply these constraints, whilst otherwise proceeding with de- coding under their grammar and model. By eval- uating performance with and without constraints, we can directly observe the cascading impact of each error class on each the parser. We implement our procedure for the graph- based MSTparser (McDonald and Pereira, 2006) and the transition-based ZPar (Zhang and Clark, 2011) using basic Stanford dependencies over the OntoNotes 4.0 release of the WSJ Penn Treebank data. Our results show that erroneously attach- ing NPs, PPs, modifiers, and punctuation have the largest overall impact on UAS. Of those, NPs and punctuation …

Content truncated to 500 words