Contradictions and Justifications: Extensions to the Textual Entailment Task

Proceedings of ACL-08: HLT, pages 63–71, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Contradictions and Justifications: Extensions to the Textual Entailment Task Ellen M. Voorhees National Institute of Standards and Technology Gaithersburg, MD 20899-8940, USA ellen.voorhees@nist.gov Abstract The third PASCAL Recognizing Textual En- tailment Challenge (RTE-3) contained an op- tional task that extended the main entailment task by requiring a system to make three-way entailment decisions (entails, contradicts, nei- ther) and to justify its response. Contradic- tion was rare in the RTE-3 test set, occurring in only about 10% of the cases, and systems found accurately detecting it difficult. Subse- quent analysis of the results shows a test set must contain many more entailment pairs for the three-way decision task than the traditional two-way task to have equal confidence in sys- tem comparisons. Each of six human judges representing eventual end users rated the qual- ity of a justification by assigning “understand- ability” and “correctness” scores. Ratings of the same justification across judges differed significantly, signaling the need for a better characterization of the justification task. 1 Introduction The PASCAL Recognizing Textual Entailment (RTE) workshop series (see www.pascal-network. org/Challenges/RTE3/) has been a catalyst for recent research in developing systems that are able to detect when the content of one piece of text necessarily follows from the content of another piece of text (Dagan et al., 2006; Giampiccolo et al., 2007). This ability is seen as a fundamental com- ponent in the solutions for a variety of natural lan- guage problems such as question answering, sum- marization, and information extraction. In addition to the main entailment task, the most recent Chal- lenge, RTE-3, contained a second optional task that extended the main task in two ways. The fi rst exten- sion was to require systems to make three-way en- tailment decisions; the second extension was for sys- tems to return a justifi cation or explanation of how its decision was reached. In the main RTE entailment task, systems report whether the hypothesis is entailed by the text. The system responds with YES if the hypothesis is en- tailed and NO otherwise. But this binary decision conflates the case when the hypothesis actually con- tradicts the text—the two could not both be true— with simple lack of entailment. The three-way en- tailment decision task requires systems to decide whether the hypothesis is entailed by the text (YES), contradicts the text (NO), or is neither entailed by nor contradicts the text (UNKNOWN). The second extension required a system to explain why it reached its conclusion in terms suitable for an eventual end user (i.e., not system developer). Ex- planations are one way to build a user’s trust in a system, but it is not known what kinds of informa- tion must be conveyed nor how best to present that information. RTE-3 provided an opportunity to col- lect a diverse sample of explanations to begin to ex- plore these questions. This paper analyzes the extended task results, with the next section …

Content truncated to 500 words