Parser Evaluation over Local and Non-Local Deep Dependencies in a Large Corpus

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 397–408, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Parser Evaluation over Local and Non-Local Deep Dependencies in a Large Corpus Emily M. Bender♠, Dan Flickinger♥, Stephan Oepen♣, Yi Zhang♦ ♠Dept of Linguistics, University of Washington, ♥CSLI, Stanford University ♣Dept of Informatics, Universitetet i Oslo, ♦Dept of Computational Linguistics, Saarland University ebender@uw.edu, danf@stanford.edu, oe@ifi.uio.no, yzhang@coli.uni-sb.de Abstract In order to obtain a fine-grained evaluation of parser accuracy over naturally occurring text, we study 100 examples each of ten reason- ably frequent linguistic phenomena, randomly selected from a parsed version of the En- glish Wikipedia. We construct a correspond- ing set of gold-standard target dependencies for these 1000 sentences, operationalize map- pings to these targets from seven state-of-the- art parsers, and evaluate the parsers against this data to measure their level of success in identifying these dependencies. 1 Introduction The terms “deep” and “shallow” are frequently used to characterize or contrast different approaches to parsing. Inevitably, such informal notions lack a clear definition, and there is little evidence of com- munity consensus on the relevant dimension(s) of depth, let alone agreement on applicable metrics. At its core, the implied dichotomy of approaches al- ludes to differences in the interpretation of the pars- ing task. Its abstract goal, on the one hand, could be pre-processing of the linguistic signal, to enable subsequent stages of analysis. On the other hand, it could be making explicit the (complete) contribution that the grammatical form of the linguistic signal makes to interpretation, working out who did what to whom. Stereotypically, one expects correspond- ing differences in the choice of interface representa- tions, ranging from various levels of syntactic anal- ysis to logical-form representations of semantics. In this paper, we seek to probe aspects of variation in automated linguistic analysis. We make the as- sumption that an integral part of many (albeit not all) applications of parsing technology is the recovery of structural relations, i.e. dependencies at the level of interpretation. We suggest a selection of ten linguis- tic phenomena that we believe (a) occur with reason- ably high frequency in running text and (b) have the potential to shed some light on the depths of linguis- tic analysis. We quantify the frequency of these con- structions in the English Wikipedia, then annotate 100 example sentences for each phenomenon with gold-standard dependencies reflecting core proper- ties of the phenomena of interest. This gold standard is then used to estimate the recall of these dependen- cies by seven commonly used parsers, providing the basis for a qualitative discussion of the state of the art in parsing for English. In this work, we answer the call by Rimell et al. (2009) for “construction-focused parser evalua- tion”, extending and complementing their work in several respects: (i) we investigate both local and non-local dependencies which prove to be challeng- ing for many existing state-of-the-art parsers; (ii) we investigate a wider range of linguistic phenomena, each accompanied …

Content truncated to 500 words