What do we expect from Multiple-choice QA Systems?

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3547–3553 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3547 What do we expect from Multiple-choice QA Systems? Krunal Shah and Nitish Gupta and Dan Roth University of Pennsylvania {shahkr, nitishg, danroth}@seas.upenn.edu Abstract The recent success of machine learning sys- tems on various QA datasets could be inter- preted as a significant improvement in mod- els’ language understanding abilities. How- ever, using various perturbations, multiple re- cent works have shown that good performance on a dataset might not indicate performance that correlates well with human’s expectations from models that “understand" language. In this work we consider a top performing model on several Multiple Choice Question Answer- ing (MCQA) datasets, and evaluate it against a set of expectations one might have from such a model, using a series of zero-information per- turbations of the model’s inputs. Our results show that the model clearly falls short of our expectations, and motivates a modified train- ing approach that forces the model to better at- tend to the inputs. We show that the new train- ing paradigm leads to a model that performs on par with the original model while better sat- isfying our expectations.1 1 Introduction Question answering (QA) has been a prevalent for- mat for gauging advances in language understand- ing. Recent advances in contextual language mod- elling have led to impressive results on multiple NLP tasks, including on several multiple choice question answering (MCQA, depicted in Fig. 1) datasets, a particularly interesting QA task that pro- vides a flexible space of candidate answers along with a simple evaluation. However, recent work (Khashabi et al., 2016; Jia and Liang, 2017; Si et al., 2019; Gardner et al., 2019, inter alia) has questioned the interpretation of these QA successes as progress in natural language understanding. Indeed, they exhibit, in various task 1Resources for this work are available at: http://cogcomp.org/page/publication_view/913 Figure 1: An example from ARC Easy dataset (Clark et al., 2018) showing the three MCQA task inputs. settings, the brittleness of neural models to various perturbations. They also show (Kaushik and Lip- ton, 2018; Gururangan et al., 2018) how models could learn to latch on to spurious correlations in the data to achieve high performance on a given dataset. In this paper we continue this line of work with a careful analysis of the extent to which the top performing MCQA model satisfies one’s expec- tation from a model that “understands" language. We formulate the following set of (non- exhaustive) expectation principles that a MCQA model should satisfy. Monotonicity Expectation: Model performance should not drop if an incorrect option is changed to make it even less likely to be correct. Sanity Expectation: Model should perform poorly given trivially insufficient input. Reading Expectation: Model should only choose an answer that is supported by the provided con- text (and thus perform poorly in the absence of informative context). While we view the first two expectation princi- ples as necessary axioms, the third could …

Content truncated to 500 words