The Emergence of Argument Structure in Artificial Languages

Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition Mor Geva, Tomer Wolfson, Jonathan Berant School of Computer Science, Tel Aviv University, Israel Allen Institute for Artificial Intelligence {morgeva@mail,tomerwol@mail,joberant@cs}.tau.ac.il Abstract Recent efforts to create challenge benchmarks that test the abilities of natural language un- derstanding models have largely depended on human annotations. In this work, we in- troduce the ‘‘Break, Perturb, Build’’ (BPB) framework for automatic reasoning-oriented perturbation of question-answer pairs. BPB represents a question by decomposing it into the reasoning steps that are required to answer it, symbolically perturbs the decomposition, and then generates new question-answer pairs. We demonstrate the effectiveness of BPB by creating evaluation sets for three reading comprehension (RC) benchmarks, generating thousands of high-quality examples without human intervention. We evaluate a range of RC models on our evaluation sets, which reveals large performance gaps on generated exam- ples compared to the original data. Moreover, symbolic perturbations enable fine-grained analysis of the strengths and limitations of models. Last, augmenting the training data with examples generated by BPB helps close the performance gaps, without any drop on the original data distribution. 1 Introduction Evaluating natural language understanding (NLU) systems has become a fickle enterprise. While models outperform humans on standard bench- marks, they perform poorly on a multitude of distribution shifts (Jia and Liang, 2017; Naik et al., 2018; McCoy et al., 2019, inter alia). To expose such gaps, recent work has proposed to evaluate models on contrast sets (Gardner et al., 2020), or counterfactually-augmented data (Kaushik et al., 2020), where minimal but meaningful pertur- bations are applied to test examples. However, since such examples are manually written, col- lecting them is expensive, and procuring diverse perturbations is challenging (Joshi and He, 2021). Recently, methods for automatic generation of contrast sets were proposed. However, current methods are restricted to shallow surface pertur- bations (Mille et al., 2021; Li et al., 2020), specific reasoning skills (Asai and Hajishirzi, 2020), or rely on expensive annotations (Bitton et al., 2021). Thus, automatic generation of examples that test high-level reasoning abilities of models and their robustness to fine semantic distinctions remains an open challenge. In this work, we propose the ‘‘Break, Perturb, Build’’ (BPB) framework for automatic genera- tion of reasoning-focused contrast sets for read- ing comprehension (RC). Changing the high-level semantics of questions and generating question- answer pairs automatically is challenging. First, it requires extracting the reasoning path expressed in a question, in order to manipulate it. Second, it requires the ability to generate grammatical and coherent questions. In Figure 1, for example, trans- forming Q, which involves number comparison, into Q1, which requires subtraction, leads to dra- matic changes in surface form. Third, it requires an automatic method for computing the answer to the perturbed question. Our insight is that perturbing question semantics is possible when modifications are applied to a structured meaning representation, rather than to the question itself. Specifically, we represent questions with QDMR (Wolfson et al., 2020), a representation that decomposes a question into a …

Content truncated to 500 words