To Test Machine Comprehension, Start by Defining Comprehension

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7839–7859 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7839 To Test Machine Comprehension, Start by Defining Comprehension Jesse Dunietz∗, Gregory Burnham∗, Akash Bharadwaj, Owen Rambow, Jennifer Chu-Carroll, and David Ferrucci Elemental Cognition {jessed,gregb,akashb,owenr,jenniferc,davef} @elementalcognition.com Abstract Many tasks aim to measure MACHINE READ- ING COMPREHENSION (MRC), often focus- ing on question types presumed to be diffi- cult. Rarely, however, do task designers start by considering what systems should in fact comprehend. In this paper we make two key contributions. First, we argue that existing approaches do not adequately define compre- hension; they are too unsystematic about what content is tested. Second, we present a de- tailed definition of comprehension—a TEM- PLATE OF UNDERSTANDING—for a widely useful class of texts, namely short narratives. We then conduct an experiment that strongly suggests existing systems are not up to the task of narrative understanding as we define it. 1 Introduction Over the past few years, neural models (e.g., Chen et al., 2016; Devlin et al., 2019; Liu et al., 2019) have begun to match or even exceed human per- formance on MACHINE READING COMPREHEN- SION (MRC) benchmarks. In these tasks, systems demonstrate their comprehension of a passage by answering questions about it. Yet despite recent successes, MRC appears far from solved: systems continue to make basic, sometimes baffling mis- takes, and they fail to generalize to new data. Such shortcomings have motivated a flurry of new MRC tasks, each designed to confront systems with ques- tions deemed challenging for current methods. For example, tasks may ask questions requiring com- monsense reasoning (Huang et al., 2019), multi- hop reasoning (Welbl et al., 2018), or inferences based on a second passage (Lin et al., 2019). This line of research assumes that ever-more- “difficult” question-answering tasks will ultimately lead to more robust and useful reading comprehen- sion. We argue that, while the question-answering *Equal contributions. format can be a fine choice for how to test com- prehension, using difficulty as the basis for what to test is fundamentally flawed. To put it provoca- tively, the dominant MRC research paradigm is like trying to become a professional sprinter by glanc- ing around the gym and adopting any exercises that look hard. The training may end up exercising some relevant muscles, but it is far too haphazard to achieve the ultimate goal. Like athletic training, MRC tasks are not an end in themselves; ultimately, they are meant to lead to real-world applications. Current tasks may suffice for sufficiently similar applications—e.g., chatbots that look up customer questions in product docu- mentation. But many proposed NLP applications hinge on deeper comprehension. Early work (e.g., Dyer, 1982) pointed to examples like assistance with legal disputes and service contracts; more re- cent work suggests applications such as summariz- ing a patient’s clinical timeline (Jung et al., 2011). For such complex applications, machines will need to manipulate rich models of the world evoked by the text—e.g., to compare …

Content truncated to 500 words