Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8846–8863, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8846 Is Multihop QA in DIRE Condition? Measuring and Reducing Disconnected Reasoning Harsh Trivedi†∗ Niranjan Balasubramanian† Tushar Khot‡ Ashish Sabharwal‡ † Stony Brook University, Stony Brook, U.S.A. {hjtrivedi,niranjan}@cs.stonybrook.edu ‡ Allen Institute for AI, Seattle, U.S.A. {tushark,ashishs}@allenai.org Abstract Has there been real progress in multi-hop question-answering? Models often exploit dataset artifacts to produce correct answers, without connecting information across multi- ple supporting facts. This limits our ability to measure true progress and defeats the pur- pose of building multi-hop QA datasets. We make three contributions towards addressing this. First, we formalize such undesirable be- havior as disconnected reasoning across sub- sets of supporting facts. This allows develop- ing a model-agnostic probe for measuring how much any model can cheat via disconnected reasoning. Second, using a notion of con- trastive support sufficiency, we introduce an automatic transformation of existing datasets that reduces the amount of disconnected rea- soning. Third, our experiments1 suggest that there hasn’t been much progress in multifact QA in the reading comprehension setting. For a recent large-scale model (XLNet), we show that only 18 points out of its answer F1 score of 72 on HotpotQA are obtained through mul- tifact reasoning, roughly the same as that of a simpler RNN baseline. Our transformation substantially reduces disconnected reasoning (19 points in answer F1). It is complementary to adversarial approaches, yielding further re- ductions in conjunction. 1 Introduction Multi-hop question answering requires connecting and synthesizing information from multiple facts in the input text, a process we refer to as multi- fact reasoning. Prior work has, however, shown that bad reasoning models, ones that by design do not connect information from multiple facts, can achieve high scores because they can exploit spe- cific types of biases and artifacts (e.g., answer type ∗Early portion of this work was done during the first author’s internship at Allen Institute for AI. 1https://github.com/stonybrooknlp/dire Which country got independence when the cold war started? Supporting Facts (SF) The war started in 1950. The cold war started in 1947. France finally got its independence. India got independence from UK in 1947. 30 countries were involved in World War 2. Set of Facts India Disconnected Reasoning Answer India Simple Combination Input Output Answer SF No Interaction Figure 1: Example of disconnected reasoning, a form of bad multifact reasoning: Model arrives at the answer by simply combining its outputs from two subsets of the input, neither of which contains all supporting facts. (From one subset, it identifies the blue supporting fact ), the only one mentioning cold war. Independently, from the other subset, it finds the red fact ( ) as the only one mentioning a country getting independence with associated time, and returns the correct answer (In- dia). Further, it returns a simple union of the supporting facts it found over the input subsets. shortucts) in existing datasets (Min et al., 2019; Chen and Durrett, 2019). While this …

Content truncated to 500 words