“Ask Not What Textual Entailment Can Do for You...”

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1199–1208, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics “Ask not what Textual Entailment can do for You...” Mark Sammons V.G.Vinod Vydiswaran Dan Roth University of Illinois at Urbana-Champaign {mssammon|vgvinodv|danr}@illinois.edu Abstract We challenge the NLP community to par- ticipate in a large-scale, distributed effort to design and build resources for devel- oping and evaluating solutions to new and existing NLP tasks in the context of Rec- ognizing Textual Entailment. We argue that the single global label with which RTE examples are annotated is insufficient to effectively evaluate RTE system perfor- mance; to promote research on smaller, re- lated NLP tasks, we believe more detailed annotation and evaluation are needed, and that this effort will benefit not just RTE researchers, but the NLP community as a whole. We use insights from success- ful RTE systems to propose a model for identifying and annotating textual infer- ence phenomena in textual entailment ex- amples, and we present the results of a pi- lot annotation study that show this model is feasible and the results immediately use- ful. 1 Introduction Much of the work in the field of Natural Lan- guage Processing is founded on an assumption of semantic compositionality: that there are iden- tifiable, separable components of an unspecified inference process that will develop as research in NLP progresses. Tasks such as Named En- tity and coreference resolution, syntactic and shal- low semantic parsing, and information and rela- tion extraction have been identified as worthwhile tasks and pursued by numerous researchers. While many have (nearly) immediate application to real world tasks like search, many are also motivated by their potential contribution to more ambitious Natural Language tasks. It is clear that the compo- nents/tasks identified so far do not suffice in them- selves to solve tasks requiring more complex rea- soning and synthesis of information; many other tasks must be solved to achieve human-like perfor- mance on tasks such as Question Answering. But there is no clear process for identifying potential tasks (other than consensus by a sufficient num- ber of researchers), nor for quantifying their po- tential contribution to existing NLP tasks, let alone to Natural Language Understanding. Recent “grand challenges” such as Learning by Reading, Learning To Read, and Machine Reading are prompting more careful thought about the way these tasks relate, and what tasks must be solved in order to understand text sufficiently well to re- liably reason with it. This is an appropriate time to consider a systematic process for identifying semantic analysis tasks relevant to natural lan- guage understanding, and for assessing their potential impact on NLU system performance. Research on Recognizing Textual Entailment (RTE), largely motivated by a “grand challenge” now in its sixth year, has already begun to address some of the problems identified above. Tech- niques developed for RTE have now been suc- cessfully applied in the domains of Question An- swering (Harabagiu and Hickl, 2006) and Ma- chine Translation …

Content truncated to 500 words