Benchmarking Machine Reading Comprehension: A Psychological Perspective

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1592–1612 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1592 Benchmarking Machine Reading Comprehension: A Psychological Perspective Saku Sugawara1, Pontus Stenetorp2, Akiko Aizawa1 1National Institute of Informatics, 2University College London {saku,aizawa}@nii.ac.jp, p.stenetorp@cs.ucl.ac.uk Abstract Machine reading comprehension (MRC) has received considerable attention as a bench- mark for natural language understanding. However, the conventional task design of MRC lacks explainability beyond the model interpretation, i.e., reading comprehension by a model cannot be explained in human terms. To this end, this position paper provides a the- oretical basis for the design of MRC datasets based on psychology as well as psychometrics, and summarizes it in terms of the prerequisites for benchmarking MRC. We conclude that fu- ture datasets should (i) evaluate the capability of the model for constructing a coherent and grounded representation to understand context- dependent situations and (ii) ensure substan- tive validity by shortcut-proof questions and explanation as a part of the task design. 1 Introduction Evaluation of natural language understanding (NLU) is a long-standing goal in the field of artifi- cial intelligence. Machine reading comprehension (MRC) is a task that tests the ability of a machine to read and understand unstructured text and could be the most suitable task for evaluating NLU because of its generic formulation (Chen, 2018). Recently, many large-scale datasets have been proposed, and deep learning systems have achieved human-level performance for some of these datasets. However, analytical studies have shown that MRC models do not necessarily achieve human- level understanding. For example, Jia and Liang (2017) use manually crafted adversarial examples to show that successful systems are easily dis- tracted. Sugawara et al. (2020) show that a sig- nificant part of already solved questions is solvable even after shuffling the words in a sentence or drop- ping content words. These studies demonstrate that we cannot explain what type of understanding is required by the datasets and is actually acquired by models. Although benchmarking MRC is related to the intent behind questions and is critical to test hy- potheses from a top-down viewpoint (Bender and Koller, 2020), its theoretical foundation is poorly investigated in the literature. In this position paper, we examine the prerequi- sites for benchmarking MRC based on the follow- ing two questions: (i) What does reading compre- hension involve? (ii) How can we evaluate it? Our motivation is to provide a theoretical basis for the creation of MRC datasets. As Gilpin et al. (2018) indicate, interpreting the internals of a system is closely related to only the system’s architecture and is insufficient for explaining how the task is ac- complished. This is because even if the internals of models can be interpreted, we cannot explain what is measured by the datasets. Therefore, our study focuses on the explainability of the task rather than the interpretability of models. We first overview MRC and review the analytical literature that indicates that existing datasets might fail to correctly evaluate …

Content truncated to 500 words