A Framework for Evaluation of Machine Reading Comprehension Gold Standards

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5359–5369 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5359 A Framework for Evaluation of Machine Reading Comprehension Gold Standards Viktor Schlegel, Marco Valentino, Andr´e Freitas, Goran Nenadic, Riza Batista-Navarro Department of Computer Science, University of Manchester Manchester, United Kingdom {viktor.schlegel, marco.valentino, andre.freitas, gnenadic, riza.batista}@manchester.ac.uk Abstract Machine Reading Comprehension (MRC) is the task of answering a question over a paragraph of text. While neural MRC systems gain popularity and achieve noticeable performance, issues are being raised with the methodology used to establish their performance, particularly concerning the data design of gold standards that are used to evaluate them. There is but a limited understanding of the challenges present in this data, which makes it hard to draw comparisons and formulate reliable hypotheses. As a first step towards alleviating the problem, this paper proposes a unifying framework to systematically investigate the present linguistic features, required reasoning and background knowledge and factual correctness on one hand, and the presence of lexical cues as a lower bound for the requirement of understanding on the other hand. We propose a qualitative annotation schema for the first and a set of approximative metrics for the latter. In a first application of the framework, we analyse modern MRC gold standards and present our findings: the absence of features that contribute towards lexical ambiguity, the varying factual correctness of the expected answers and the presence of lexical cues, all of which potentially lower the reading comprehension complexity and quality of the evaluation data. Keywords: Machine Reading Comprehension, Question Answering, Evaluation Methodology, Annotation Schema 1. Introduction There is a recent spark of interest in the task of Ques- tion Answering (QA) over unstructured textual data, also referred to as Machine Reading Comprehension (MRC). This is mostly due to wide-spread success of advances in various facets of deep learning related research, such as novel architectures (Vaswani et al., 2017; Sukhbaatar et al., 2015) that allow for efficient optimisation of neural net- works consisting of multiple layers, hardware designed for deep learning purposes12 and software frameworks (Abadi et al., 2016; Paszke et al., 2017) that allow efficient devel- opment and testing of novel approaches. These factors en- able researchers to produce models that are pre-trained on large scale corpora and provide contextualised word rep- resentations (Peters et al., 2018) that are shown to be a vital component towards solutions for a variety of natural language understanding tasks, including MRC (Devlin et al., 2019). Another important factor that led to the recent success in MRC-related tasks is the widespread availabil- ity of various large datasets, e.g., SQuAD (Rajpurkar et al., 2016), that provide sufficient examples for optimising sta- tistical models. The combination of these factors yields no- table results, even surpassing human performance (Lan et al., 2020). MRC is a generic task format that can be used to probe for various natural language understanding capabilities (Gard- ner et al., 2019). Therefore it is …

Content truncated to 500 words