Addressing Limited Data for Textual Entailment Across Domains

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1245–1255, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Addressing Limited Data for Textual Entailment Across Domains Chaitanya Shivade∗ Preethi Raghavan† and Siddharth Patwardhan† ∗Department of Computer Science and Engineering, The Ohio State University, Columbus, OH 43210 shivade@cse.ohio-state.edu †IBM T. J. Watson Research Center, 1101 Kitchawan Road, Yorktown Heights, NY 10598 {praghav,siddharth}@us.ibm.com Abstract We seek to address the lack of labeled data (and high cost of annotation) for textual entailment in some domains. To that end, we first create (for experimental purposes) an entailment dataset for the clinical do- main, and a highly competitive supervised entailment system, ENT, that is effective (out of the box) on two domains. We then explore self-training and active learn- ing strategies to address the lack of la- beled data. With self-training, we success- fully exploit unlabeled data to improve over ENT by 15% F-score on the newswire domain, and 13% F-score on clinical data. On the other hand, our active learning ex- periments demonstrate that we can match (and even beat) ENT using only 6.6% of the training data in the clinical domain, and only 5.8% of the training data in the newswire domain. 1 Introduction Textual entailment is the task of automatically de- termining whether a natural language hypothesis can be inferred from a given piece of natural lan- guage text. The RTE challenges (Bentivogli et al., 2009; Bentivogli et al., 2011) have spurred considerable research in textual entailment over newswire data. This, along with the availability of large-scale datasets labeled with entailment in- formation (Bowman et al., 2015), has resulted in a variety of approaches for textual entailment recog- nition. ∗This work was conducted during an internship at IBM A variation of this task, dubbed textual entail- ment search, has been the focus of RTE-5 and sub- sequent challenges, where the goal is to find all sentences in a corpus that entail a given hypoth- esis. The mindshare created by those challenges and the availability of the datasets has spurred many creative solutions to this problem. How- ever, the evaluations have been restricted primarily to these datasets, which are in the newswire do- main. Thus, much of the existing state-of-the-art research has focused on solutions that are effective in this domain. It is easy to see though, that entailment search has potential applications in other domains too. For instance, in the clinical domain we imagine entailment search can be applied for clinical trial matching as one example. Inclusion criteria for a clinical trial (for e.g., patient is a smoker) be- come the hypotheses, and the patient’s electronic health records are the text for entailment search. Clearly, an effective textual entailment search sys- tem could possibly one day fully automate clinical trial matching. Developing an entailment system that works well in the clinical domain and, thus, automates this matching process, requires lots of labeled data, which is extremely scant in the clinical do- main. Generating such a dataset …

Content truncated to 500 words