Learning to recognize features of valid textual entailments

Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 41–48, New York, June 2006. c⃝2006 Association for Computational Linguistics Learning to recognize features of valid textual entailments Bill MacCartney, Trond Grenager, Marie-Catherine de Marneffe, Daniel Cer, and Christopher D. Manning Computer Science Department Stanford University Stanford, CA 94305 {wcmac, grenager, mcdm, cerd, manning}@cs.stanford.edu Abstract This paper advocates a new architecture for tex- tual inference in which finding a good alignment is separated from evaluating entailment. Current ap- proaches to semantic inference in question answer- ing and textual entailment have approximated the entailment problem as that of computing the best alignment of the hypothesis to the text, using a lo- cally decomposable matching score. We argue that there are significant weaknesses in this approach, including flawed assumptions of monotonicity and locality. Instead we propose a pipelined approach where alignment is followed by a classification step, in which we extract features representing high-level characteristics of the entailment prob- lem, and pass the resulting feature vector to a statis- tical classifier trained on development data. We re- port results on data from the 2005 Pascal RTE Chal- lenge which surpass previously reported results for alignment-based systems. 1 Introduction During the last five years there has been a surge in work which aims to provide robust textual inference in arbitrary domains about which the system has no expertise. The best-known such work has occurred within the field of question answering (Pasca and Harabagiu, 2001; Moldovan et al., 2003); more re- cently, such work has continued with greater focus in addressing the PASCAL Recognizing Textual En- tailment (RTE) Challenge (Dagan et al., 2005) and within the U.S. Government AQUAINT program. Substantive progress on this task is key to many text and natural language applications. If one could tell that Protestors chanted slogans opposing a free trade agreement was a match for people demonstrat- ing against free trade, then one could offer a form of semantic search not available with current keyword- based search. Even greater benefits would flow to richer and more semantically complex NLP tasks. Because full, accurate, open-domain natural lan- guage understanding lies far beyond current capa- bilities, nearly all efforts in this area have sought to extract the maximum mileage from quite lim- ited semantic representations. Some have used sim- ple measures of semantic overlap, but the more in- teresting work has largely converged on a graph- alignment approach, operating on semantic graphs derived from syntactic dependency parses, and using a locally-decomposable alignment score as a proxy for strength of entailment. (Below, we argue that even approaches relying on weighted abduction may be seen in this light.) In this paper, we highlight the fundamental semantic limitations of this type of ap- proach, and advocate a multi-stage architecture that addresses these limitations. The three key limita- tions are an assumption of monotonicity, an assump- tion of locality, and a confounding of alignment and evaluation of entailment. We focus on the PASCAL RTE data, examples from which …

Content truncated to 500 words