Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6541 - 6566 December 7-11, 2022 ©2022 Association for Computational Linguistics Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts Harsh Trivedi†∗ Niranjan Balasubramanian‡ †Stony Brook University Stony Brook, U.S.A. {hjtrivedi,niranjan}@cs.stonybrook.edu Tushar Khot‡ Ashish Sabharwal‡ ‡Allen Institute for AI Seattle, U.S.A. {tushark,ashishs}@allenai.org Abstract Question-answering datasets require a broad set of reasoning skills. We show how to use ques- tion decompositions to teach language mod- els these broad reasoning skills in a robust fashion. Specifically, we use widely avail- able QDMR representations to programmati- cally create hard-to-cheat synthetic contexts for real questions in six multi-step reasoning datasets. These contexts are carefully designed to avoid common reasoning shortcuts preva- lent in real contexts that prevent models from learning the right skills. This results in a pre- training dataset, named TeaBReaC, containing 525K multi-step questions (with associated for- mal programs) covering about 900 reasoning patterns. We show that pretraining standard language models (LMs) on TeaBReaC before fine-tuning them on target datasets improves their performance by up to 13 F1 points across 4 multi-step QA datasets, with up to 21 point gain on more complex questions. The result- ing models also demonstrate higher robustness, with a 5-8 F1 point improvement on two con- trast sets. Furthermore, TeaBReaC pretrain- ing substantially improves model performance and robustness even when starting with numer- ate LMs pretrained using recent methods (e.g., PReasM, POET). Our work thus shows how to effectively use decomposition-guided contexts to robustly teach multi-step reasoning.1 1 Introduction Multi-step Question Answering (QA) is a complex problem that requires a wide variety of reasoning skills. In addition to basic reading comprehen- sion (RC), models must connect multiple pieces of information, sometimes employ numerical and other forms of discrete reasoning, and compose these skills as needed for the question. However, ∗The work was done during the first author’s internship at the Allen Institute for AI. 1Code and data available at https://github.com/ stonybrooknlp/teabreac. DROP, HotpotQA ... ComQA, Spider, ATIS Q: From what yard-line did Shayne Graham kick two field goals? Q: From what yard- line did Shayne ... QDMR Decomposition Synthetic Context same question Constructed by grounding QDMR predicates Question - Decomposition Pair TeaBReaC Instance Repeat N Times Sample Reasoning Pattern Questions with Decompositions Sample (Question, Decomposition) Typed Program Executable (e.g. in Python) Figure 1: TEABREAC Dataset Construction: We leverage widely available question decomposition anno- tations (QDMRs) for real questions from a broad range of datasets to carefully construct synthetic contexts such that answering the resulting question requires proper multi-step reasoning. These questions are further re- balanced to help teach a broad set of reasoning skills. even though questions in multi-step datasets often cover a broad range of interesting reasoning pat- terns, most questions follow only a few patterns, which is what models trained on these datasets nat- urally focus on. Moreover, the contexts occurring in existing RC datasets often contain artifacts and reasoning shortcuts (Min et al., 2019a; Chen and Durrett, 2019; Trivedi …

Content truncated to 500 words