Weakly- and Semi-supervised Evidence Extraction

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3965–3970 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3965 Weakly- and Semi-supervised Evidence Extraction Danish Pruthi, Bhuwan Dhingra, Graham Neubig, Zachary C. Lipton Carnegie Mellon University Pittsburgh, USA {ddanish, bdhingra, gneubig, zlipton}@cs.cmu.edu Abstract For many prediction tasks, stakeholders desire not only predictions but also supporting evi- dence that a human can use to verify its cor- rectness. However, in practice, evidence an- notations may only be available for a minority of training examples (if available at all). In this paper, we propose new methods to com- bine few evidence annotations (strong semi- supervision) with abundant document-level la- bels (weak supervision) for the task of evi- dence extraction. Evaluating on two classifi- cation tasks that feature evidence annotations, we find that our methods outperform baselines adapted from the interpretability literature to our task. Our approach yields gains with as few as hundred evidence annotations.1 1 Introduction Despite the success of deep learning for count- less prediction tasks, practitioners often desire that these models not only be accurate but also pro- vide interpretations or explanations (Caruana et al., 2015; Weld and Bansal, 2019). Unfortunately, these terms lack precise meaning, and across pa- pers, such explanations purport to address a wide spectrum of desiderata, and it seems unlikely any one method could address them all (Lipton, 2018). In both computer vision (Ribeiro et al., 2016; Si- monyan et al., 2013) and natural language process- ing (Lei et al., 2016; Lehman et al., 2019), pro- posed explanation methods often take the form of highlighting salient features of the input. These so- called local explanations are intended to highlight features that elucidate “the reasons behind predic- tions” (Ribeiro et al., 2016). However, this charac- terization of the problem remains under-specified. 1Code and datasets to reproduce our work are available at: https://github.com/danishpruthi/ evidence-extraction. In this paper, we instead focus on supplement- ing predictions with evidence, which we define as information that gives users the ability to quickly verify the correctness of predictions. Fortunately, for many problems, a localized portion of the input is sufficient to validate the predicted label. In a large image, a small patch of an image containing a hamster may be sufficient to render the “hamster” label applicable. Similarly, in a long clinical note, a single sentence may suffice to confirm a predicted diagnosis. This ability to verify results engenders trust among users, and increases adoption of the machine learning systems (Dzindolet et al., 2003; Herlocker et al., 2000; Ribeiro et al., 2016). In Table 1, we outline the characteristic differences between local explanations and evidence. Thus motivated, we cast our problem as learn- ing to extract evidence using both strong and weak supervision. The former takes the form of explicit, but scarce, manual annotations of evidence seg- ments, whereas the latter is provided by documents and their class labels which we assume are rela- tively abundant.2 In the extreme case where evi- dence annotations are available for all examples, …

Content truncated to 500 words