English Machine Reading Comprehension Datasets: A Survey

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8784–8804 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8784 English Machine Reading Comprehension Datasets: A Survey Daria Dzendzik ADAPT Centre Dublin City University Dublin, Ireland daria.dzendzik @adaptcentre.ie Carl Vogel School of Computer Science and Statistics Trinity College Dublin the University of Dublin Dublin, Ireland vogel@tcd.ie Jennifer Foster School of Computing Dublin City University Dublin, Ireland jennifer.foster @dcu.ie Abstract This paper surveys 60 English Machine Read- ing Comprehension datasets, with a view to providing a convenient resource for other re- searchers interested in this problem. We cat- egorize the datasets according to their ques- tion and answer form and compare them across various dimensions including size, vocabulary, data source, method of creation, human perfor- mance level, and first question word. Our anal- ysis reveals that Wikipedia is by far the most common data source and that there is a relative lack of why, when, and where questions across datasets. 1 Introduction Figure 1: English MRC datasets released per year Reading comprehension is often tested by mea- suring a person or system’s ability to answer ques- tions on a given text. Machine reading compre- hension (MRC) datasets have proliferated in recent years, particularly for the English language – see Fig. 1. The aim of this paper is to make sense of this landscape by providing as extensive as possi- ble a survey of English MRC datasets. The survey has been carried out with the following audience in mind: (1) those who are new to the field and would like to get a concise yet informative overview of English MRC datasets; (2) those who are planning to create a new MRC dataset; (3) MRC system developers, interested in designing the appropriate architecture for a particular dataset, choosing ap- propriate datasets for a particular architecture, or finding compatible datasets for use in transfer or joint learning. Our survey takes a mostly structured form, with the following information presented for each dataset: size, data source, creation method, hu- man performance level, whether the dataset has been “solved”, availability of a leaderboard, the most frequent first question token, and whether the dataset is publicly available. We also categorise each dataset by its question/answer type. Our study contributes to the field as follows: 1. it describes and teases apart the ways in which MRC datasets can vary according to their question and answer types; 2. it provides analysis in table and figure for- mat to facilitate easy comparison between datasets; 3. by providing a systematic comparison, and by reporting the “solvedness” status of a dataset, it brings the attention of the community to less popular and relatively understudied datasets; 4. it contains per-dataset statistics such as number of instances, average ques- tion/passage/answer length, vocabulary size and text domain which can be used to estimate the computational requirements for training an MRC system. 2 Question, Answer, and Passage Types All MRC datasets in this survey have three compo- nents: passage, question, and answer.1 We …

Content truncated to 500 words