Tell Me Why: Using Question Answering as Distant Supervision for Answer Justification

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 69–79, Vancouver, Canada, August 3 - August 4, 2017. c⃝2017 Association for Computational Linguistics Tell Me Why: Using Question Answering as Distant Supervision for Answer Justification Rebecca Sharp*, Mihai Surdeanu*, Peter Jansen*, Marco A. Valenzuela-Esc´arcega*, Peter Clark† and Michael Hammond* *University of Arizona †Allen Institute for Artificial Intelligence *{bsharp, msurdeanu, pajansen, marcov, hammond}@email.arizona.edu †peterc@allenai.org Abstract For many applications of question answer- ing (QA), being able to explain why a given model chose an answer is critical. However, the lack of labeled data for an- swer justifications makes learning this dif- ficult and expensive. Here we propose an approach that uses answer ranking as dis- tant supervision for learning how to select informative justifications, where justifica- tions serve as inferential connections be- tween the question and the correct answer while often containing little lexical over- lap with either. We propose a neural net- work architecture for QA that reranks an- swer justifications as an intermediate (and human-interpretable) step in answer selec- tion. Our approach is informed by a set of features designed to combine both learned representations and explicit features to capture the connection between questions, answers, and answer justifications. We show that with this end-to-end approach we are able to significantly improve upon a strong IR baseline in both justification ranking (+9% rated highly relevant) and answer selection (+6% P@1). 1 Introduction Developing interpretable machine learning (ML) models, that is, models where a human user can understand what the model is learning, is consid- ered by many to be crucial for ensuring usabil- ity and accelerating progress (Craven and Shav- lik, 1996; Kim et al., 2015; Letham et al., 2015; Ribeiro et al., 2016). For many applications of question answering (QA), i.e., finding short an- swers to natural language questions, simply pro- viding an answer is not sufficient. A complete Question: Which of these is a response to an internal stimulus? (A) A sunflower turns to face the rising sun. (B) A cucumber tendril wraps around a wire. (C) A pine tree knocked sideways in a landslide grows up- ward in a bend. (D) Guard cells of a tomato plant leaf close when there is little water in the roots . Justification: Plants rely on hormones to send signals within the plant in order to respond to internal stimuli such as a lack of water or nutrients. Table 1: Example of an 8th grade science question with a justification for the correct answer. Note the lack of direct lexical overlap present between the justification and the cor- rect answer, demonstrating the difficulty of the task of finding justifications using traditional distant supervision methods. approach must be interpretable, i.e., able to ex- plain why an answer is correct. For example, in the medical domain, a QA approach that an- swers treatment questions would not be trusted if the treatment recommendation is not explained in terms that can be understood by the human user. One approach to interpreting complex …

Content truncated to 500 words