Monte Carlo Syntax Marginals for Exploring and Using Dependency Parses

Proceedings of NAACL-HLT 2018, pages 917–928 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Monte Carlo Syntax Marginals for Exploring and Using Dependency Parses Katherine A. Keith, Su Lin Blodgett, and Brendan O’Connor College of Information and Computer Sciences University of Massachusetts Amherst {kkeith,blodgett,brenocon}@cs.umass.edu Abstract Dependency parsing research, which has made significant gains in recent years, typically fo- cuses on improving the accuracy of single- tree predictions. However, ambiguity is inher- ent to natural language syntax, and communi- cating such ambiguity is important for error analysis and better-informed downstream ap- plications. In this work, we propose a tran- sition sampling algorithm to sample from the full joint distribution of parse trees defined by a transition-based parsing model, and demon- strate the use of the samples in probabilistic dependency analysis. First, we define the new task of dependency path prediction, inferring syntactic substructures over part of a sentence, and provide the first analysis of performance on this task. Second, we demonstrate the use- fulness of our Monte Carlo syntax marginal method for parser error analysis and calibra- tion. Finally, we use this method to propa- gate parse uncertainty to two downstream in- formation extraction applications: identifying persons killed by police and semantic role as- signment.1 1 Introduction Dependency parsers typically predict a single tree for a sentence to be used in downstream applica- tions, and most work on dependency parsers seeks to improve accuracy of such single-tree predic- tions. Despite tremendous gains in the last few decades of parsing research, accuracy is far from perfect—but perfect accuracy may be impossible since syntax models by themselves do not incorpo- rate the discourse, pragmatic, or world knowledge necessary to resolve many ambiguities. In fact, although relatively unexamined, sub- stantial ambiguity already exists within commonly used discriminative probabilistic parsing models, 1Supporting code available at https://github.com/slanglab/ transition sampler Figure 1: Example of a sentence with inherent ambigu- ity. Top: output from a greedy parser. Bottom: edge marginal probabilities from 100 samples in parenthe- ses. which define a parse forest—a probability distri- bution p(y | x) over possible dependency trees y ∈ Y(x) for an input sentence x. For example, the top of Figure 1 shows the pre- dicted parse y(greedy) from such a parser (Chen and Manning, 2014), which resolves a preposi- tional (PP) attachment ambiguity in one manner; this prediction was selected by a standard greedy transition-based algorithm (§2.1). However, the bottom of Figure 1 shows marginal probabilities of individual (relation, governor, child) edges un- der this same model. These denote our estimated probabilities, across all possible parse structures, that a pair of words are connected with a particu- lar relation (§2.4). For example, the two different PP attachment readings both exist within this parse forest with marginal probabilities p( nmod(saw2, telescope7) | x) = 0.72 (1) p( nmod(man4, telescope7) | x) = 0.28, (2) where (1) implies she used a telescope to see the man, and (2) implies she saw a man who had a telescope. These types …

Content truncated to 500 words