Complexity Metrics in an Incremental Right-Corner Parser

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1189–1198, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Complexity Metrics in an Incremental Right-corner Parser Stephen Wu Asaf Bachrach† Carlos Cardenas∗ William Schuler◦ Department of Computer Science, University of Minnesota † Unit de Neuroimagerie Cognitive INSERM-CEA ∗ Department of Brain & Cognitive Sciences, Massachussetts Institute of Technology ◦ University of Minnesota and The Ohio State University swu@cs.umn.edu †asaf@mit.edu ∗cardenas@mit.edu ◦schuler@ling.ohio-state.edu Abstract Hierarchical HMM (HHMM) parsers make promising cognitive models: while they use a bounded model of working memory and pursue incremental hypothe- ses in parallel, they still achieve parsing accuracies competitive with chart-based techniques. This paper aims to validate that a right-corner HHMM parser is also able to produce complexity metrics, which quantify a reader’s incremental difficulty in understanding a sentence. Besides defining standard metrics in the HHMM framework, a new metric, embedding difference, is also proposed, which tests the hypothesis that HHMM store elements represents syntactic working memory. Results show that HHMM surprisal outperforms all other evaluated metrics in predicting reading times, and that embedding difference makes a significant, independent contribution. 1 Introduction Since the introduction of a parser-based calcula- tion for surprisal by Hale (2001), statistical tech- niques have been become common as models of reading difficulty and linguistic complexity. Sur- prisal has received a lot of attention in recent lit- erature due to nice mathematical properties (Levy, 2008) and predictive ability on eye-tracking move- ments (Demberg and Keller, 2008; Boston et al., 2008a). Many other complexity metrics have been suggested as mutually contributing to reading difficulty; for example, entropy reduction (Hale, 2006), bigram probabilities (McDonald and Shill- cock, 2003), and split-syntactic/lexical versions of other metrics (Roark et al., 2009). A parser-derived complexity metric such as sur- prisal can only be as good (empirically) as the model of language from which it derives (Frank, 2009). Ideally, a psychologically-plausible lan- guage model would produce a surprisal that would correlate better with linguistic complexity. There- fore, the specification of how to encode a syntac- tic language model is of utmost importance to the quality of the metric. However, it is difficult to quantify linguis- tic complexity and reading difficulty. The two commonly-used empirical quantifications of read- ing difficulty are eye-tracking measurements and word-by-word reading times; this paper uses read- ing times to find the predictiveness of several parser-derived complexity metrics. Various fac- tors (i.e., from syntax, semantics, discourse) are likely necessary for a full accounting of linguis- tic complexity, so current computational models (with some exceptions) narrow the scope to syn- tactic or lexical complexity. Three complexity metrics will be calculated in a Hierarchical Hidden Markov Model (HHMM) parser that recognizes trees in right-corner form (the left-right dual of left-corner form). This type of parser performs competitively on standard pars- ing tasks (Schuler et al., 2010); also, it reflects plausible accounts of human language processing as incremental (Tanenhaus et al., 1995; Brants and Crocker, 2000), as considering hypotheses proba- bilistically in parallel (Dahan and …

Content truncated to 500 words