Finding Generalizable Evidence by Learning to Convince Q&A Models

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2402–2411, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2402 Finding Generalizable Evidence by Learning to Convince Q&A Models Ethan Perez† Siddharth Karamcheti‡ Rob Fergus†‡ Jason Weston†‡ Douwe Kiela‡ Kyunghyun Cho†‡? †New York University, ‡Facebook AI Research, ?CIFAR Azrieli Global Scholar perez@nyu.edu Abstract We propose a system that finds the strongest supporting evidence for a given answer to a question, using passage-based question- answering (QA) as a testbed. We train evi- dence agents to select the passage sentences that most convince a pretrained QA model of a given answer, if the QA model received those sentences instead of the full passage. Rather than finding evidence that convinces one model alone, we find that agents select ev- idence that generalizes; agent-chosen evidence increases the plausibility of the supported an- swer, as judged by other QA models and hu- mans. Given its general nature, this approach improves QA in a robust manner: using agent- selected evidence (i) humans can correctly an- swer questions with only ⇠20% of the full passage and (ii) QA models can generalize to longer passages and harder questions. 1 Introduction There is great value in understanding the fun- damental nature of a question (Chalmers, 2015). Distilling the core of an issue, however, is time- consuming. Finding the correct answer to a given question may require reading large volumes of text or understanding complex arguments. Here, we examine if we can automatically discover the un- derlying properties of problems such as question answering by examining how machine learning models learn to solve that task. We examine this question in the context of passage-based question-answering (QA). Inspired by work in interpreting neural networks (Lei et al., 2016), we have agents find a subset of the passage (i.e., supporting evidence) that maximizes a QA model’s probability of a particular answer. Each agent (one agent per answer) finds the sentences that a QA model regards as strong evidence for its answer, using either exhaustive search or learned prediction. Figure 1 shows an example. Figure 1: Evidence agents quote sentences from the passage to convince a question-answering judge model of an answer. To examine to what extent evidence is general and independent of the model, we evaluate if hu- mans and other models find selected evidence to be valid support for an answer too. We find that, when provided with evidence selected by a given agent, both humans and models favor that agent’s answer over other answers. When human evalu- ators read an agent’s selected evidence in lieu of the full passage, humans tend to select the agent- supported answer. Given that this approach appears to capture some general, underlying properties of the prob- lem, we examine if evidence agents can be used to assist human QA and to improve generalization of other QA models. We find that humans can accu- rately answer questions on QA benchmarks, …

Content truncated to 500 words