Evaluating Induced CCG Parsers on Grounded Semantic Parsing

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2022–2027, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Evaluating Induced CCG Parsers on Grounded Semantic Parsing Yonatan Bisk1∗ Siva Reddy2∗ John Blitzer3 Julia Hockenmaier4 Mark Steedman2 1ISI, University of Southern California 2ILCC, School of Informatics, University of Edinburgh 3Google, Mountain View 4Department of Computer Science, University of Illinois at Urbana-Champaign ybisk@isi.edu, siva.reddy@ed.ac.uk, blitzer@google.com, juliahmr@illinois.edu, steedman@inf.ed.ac.uk, Abstract We compare the effectiveness of four differ- ent syntactic CCG parsers for a semantic slot- filling task to explore how much syntactic su- pervision is required for downstream seman- tic analysis. This extrinsic, task-based evalua- tion also provides a unique window into the se- mantics captured (or missed) by unsupervised grammar induction systems. 1 Introduction The past several years have seen significant progress in unsupervised grammar induction (Carroll and Charniak, 1992; Yuret, 1998; Klein and Manning, 2004; Spitkovsky et al., 2010; Garrette et al., 2015; Bisk and Hockenmaier, 2015). But how useful are unsupervised syntactic parsers for downstream NLP tasks? What phenomena are they able to capture, and where would additional annotation be required? Instead of standard intrinsic evaluations – attachment scores that depend strongly on the particular anno- tation styles of the gold treebank – we examine the utility of unsupervised and weakly supervised parsers for semantics. We perform an extrinsic evaluation of unsupervised and weakly supervised CCG parsers on a grounded semantic parsing task that will shed light on the extent to which these systems recover seman- tic information. We focus on English to perform a direct comparison with supervised parsers (although unsupervised or weakly supervised approaches are likely to be most beneficial for domains or languages where supervised parsers are not available). ∗Equal contribution Specifically, we evaluate different parsing scenar- ios with varying amounts of supervision. These are designed to shed light on the question of how well syntactic knowledge correlates with performance on a semantic evaluation. We evaluate the following sce- narios (all of which assume POS-tagged input): 1) no supervision; 2) a lexicon containing words mapped to CCG categories; 3) a lexicon containing POS tags mapped to CCG categories; 4) sentences annotated with CCG derivations (i.e., fully supervised). Our evaluation reveals which constructions are problem- atic for unsupervised parsers (and annotation efforts should focus on). Our results indicate that unsuper- vised syntax is useful for semantics, while a simple semi-supervised parser outperforms a fully unsuper- vised approach, and could hence be a viable option for low resource languages. 2 CCG Intrinsic Evaluations CCG (Steedman, 2000) is a lexicalized formalism in which words are assigned syntactic types, also known as supertags, encoding subcategorization informa- tion. Consider the sentence Google acquired Nest in 2014, and its CCG derivations shown in Figure 1. In (a) and (b), the supertag of acquired, (S\NP)/NP, indicates that it has two arguments, and the preposi- tional phrase in 2014 is an adjunct, whereas in (c) the supertag ((S\NP)/PP)/NP indicates acquired has three arguments including the prepositional phrase. In (a) and …

Content truncated to 500 words