Improving NLP through Marginalization of Hidden Syntactic Structure

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 810–820, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Improving NLP through Marginalization of Hidden Syntactic Structure Jason Naradowsky, Sebastian Riedel, and David A. Smith Department of Computer Science University of Massachusetts Amherst Amherst, MA, 01003, U.S.A. {narad, riedel, dasmith}@cs.umass.edu Abstract Many NLP tasks make predictions that are in- herently coupled to syntactic relations, but for many languages the resources required to pro- vide such syntactic annotations are unavail- able. For others it is unclear exactly how much of the syntactic annotations can be ef- fectively leveraged with current models, and what structures in the syntactic trees are most relevant to the current task. We propose a novel method which avoids the need for any syntactically annotated data when predicting a related NLP task. Our method couples latent syntactic representa- tions, constrained to form valid dependency graphs or constituency parses, with the predic- tion task via specialized factors in a Markov random field. At both training and test time we marginalize over this hidden structure, learn- ing the optimal latent representations for the problem. Results show that this approach pro- vides significant gains over a syntactically un- informed baseline, outperforming models that observe syntax on an English relation extrac- tion task, and performing comparably to them in semantic role labeling. 1 Introduction Many NLP tasks are inherently tied to syntax, and state-of-the-art solutions to these tasks often rely on syntactic annotations as either a source for useful features (Zhang et al., 2006, path features in relation extraction) or as a scaffolding upon which a more narrow, specialized classification can occur (as of- ten done in semantic role labeling). This decou- pling of the end task from its intermediate repre- sentation is sometimes known as the two-stage ap- proach (Chang et al., 2010) and comes with several drawbacks. Most notably this decomposition pro- hibits the learning method from utilizing the labels from the end task when predicting the intermediate representation, a structure which must have some correlation to the end task to provide any benefit. Relying on intermediate representations that are specifically syntactic in nature introduces its own unique set of problems. Large amounts of syntac- tically annotated data is difficult to obtain, costly to produce, and often tied to a particular domain that may vary greatly from that of the desired end task. Additionally, current systems often utilize only a small amount of the annotation for any particular task. For instance, performing named entity recogni- tion (NER) jointly with constituent parsing has been shown to improve performance on both tasks, but the only aspect of the syntax which is leveraged by the NER component is the location of noun phrases (Finkel and Manning, 2009). By instead discover- ing a latent representation jointly with the end task we address all of these concerns, alleviating the need for any syntactic annotations, while simultaneously attempting to learn a latent syntax relevant …

Content truncated to 500 words