Effectively Using Syntax for Recognizing False Entailment

Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 33–40, New York, June 2006. c⃝2006 Association for Computational Linguistics Effectively Using Syntax for Recognizing False Entailment Rion Snow Computer Science Department Stanford University Stanford, CA 94305 rion@cs.stanford.edu Lucy Vanderwende and Arul Menezes Microsoft Research One Microsoft Way Redmond, WA 98027 {lucyv,arulm}@microsoft.com Abstract Recognizing textual entailment is a chal- lenging problem and a fundamental com- ponent of many applications in natural language processing. We present a novel framework for recognizing textual entail- ment that focuses on the use of syntactic heuristics to recognize false entailment. We give a thorough analysis of our sys- tem, which demonstrates state-of-the-art performance on a widely-used test set. 1 Introduction Recognizing the semantic equivalence of two frag- ments of text is a fundamental component of many applications in natural language processing. Recog- nizing textual entailment, as formulated in the recent PASCAL Challenge 1, is the problem of determining whether some text sentence T entails some hypothe- sis sentence H. The motivation for this formulation was to iso- late and evaluate the application-independent com- ponent of semantic inference shared across many ap- plication areas, reflected in the division of the PAS- CAL RTE dataset into seven distinct tasks: Informa- tion Extraction (IE), Comparable Documents (CD), Reading Comprehension (RC), Machine Translation (MT), Information Retrieval (IR), Question Answer- ing (QA), and Paraphrase Acquisition (PP). 1http://www.pascal-network.org/Challenges/RTE. The ex- amples given throughout this paper are from the first PASCAL RTE dataset, described in Section 6. The RTE problem as presented in the PASCAL RTE dataset is particularly attractive in that it is a reasonably simple task for human annotators with high inter-annotator agreement (95.1% in one inde- pendent labeling (Bos and Markert, 2005)), but an extremely challenging task for automated systems. The highest accuracy systems on the RTE test set are still much closer in performance to a random baseline accuracy of 50% than to the inter-annotator agreement. For example, two high-accuracy systems are those described in (Tatu and Moldovan, 2005), achieving 60.4% accuracy with no task-specific in- formation, and (Bos and Markert, 2005), which achieves 61.2% task-dependent accuracy, i.e. when able to use the specific task labels as input. Previous systems for RTE have attempted a wide variety of strategies. Many previous approaches have used a logical form representation of the text and hypothesis sentences, focusing on deriving a proof by which one can infer the hypothesis logical form from the text logical form (Bayer et al., 2005; Bos and Markert, 2005; Raina et al., 2005; Tatu and Moldovan, 2005). These papers often cite that a ma- jor obstacle to accurate theorem proving for the task of textual entailment is the lack of world knowledge, which is frequently difficult and costly to obtain and encode. Attempts have been made to remedy this deficit through various techniques, including model- building (Bos and Markert, 2005) and the addition of semantic axioms (Tatu and Moldovan, 2005). Our system diverges from previous approaches most strongly by focusing …

Content truncated to 500 words