Instance-based Evaluation of Entailment Rule Acquisition

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 456–463, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Instance-based Evaluation of Entailment Rule Acquisition Idan Szpektor, Eyal Shnarch, Ido Dagan Dept. of Computer Science Bar Ilan University Ramat Gan, Israel {szpekti,shey,dagan}@cs.biu.ac.il Abstract Obtaining large volumes of inference knowl- edge, such as entailment rules, has become a major factor in achieving robust seman- tic processing. While there has been sub- stantial research on learning algorithms for such knowledge, their evaluation method- ology has been problematic, hindering fur- ther research. We propose a novel evalua- tion methodology for entailment rules which explicitly addresses their semantic proper- ties and yields satisfactory human agreement levels. The methodology is used to compare two state of the art learning algorithms, ex- posing critical issues for future progress. 1 Introduction In many NLP applications, such as Question An- swering (QA) and Information Extraction (IE), it is crucial to recognize that a particular target mean- ing can be inferred from different text variants. For example, a QA system needs to identify that “As- pirin lowers the risk of heart attacks” can be inferred from “Aspirin prevents heart attacks” in order to an- swer the question “What lowers the risk of heart at- tacks?”. This type of reasoning has been recognized as a core semantic inference task by the generic tex- tual entailment framework (Dagan et al., 2006). A major obstacle for further progress in seman- tic inference is the lack of broad-scale knowledge- bases for semantic variability patterns (Bar-Haim et al., 2006). One prominent type of inference knowl- edge representation is inference rules such as para- phrases and entailment rules. We define an entail- ment rule to be a directional relation between two templates, text patterns with variables, e.g. ‘X pre- vent Y → X lower the risk of Y ’. The left-hand- side template is assumed to entail the right-hand- side template in certain contexts, under the same variable instantiation. Paraphrases can be viewed as bidirectional entailment rules. Such rules capture basic inferences and are used as building blocks for more complex entailment inference. For example, given the above rule, the answer “Aspirin” can be identified in the example above. The need for large-scale inference knowledge- bases triggered extensive research on automatic ac- quisition of paraphrase and entailment rules. Yet the current precision of acquisition algorithms is typ- ically still mediocre, as illustrated in Table 1 for DIRT (Lin and Pantel, 2001) and TEASE (Szpek- tor et al., 2004), two prominent acquisition algo- rithms whose outputs are publicly available. The current performance level only stresses the obvious need for satisfactory evaluation methodologies that would drive future research. The prominent approach in the literature for eval- uating rules, termed here the rule-based approach, is to present the rules to human judges asking whether each rule is correct or not. However, it is difficult to explicitly define when a learned rule should be con- sidered correct under this methodology, and this was mainly …

Content truncated to 500 words