Is the Understanding of Explicit Discourse Relations Required in Machine Reading Comprehension?

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3565–3579 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3565 Is the Understanding of Explicit Discourse Relations Required in Machine Reading Comprehension? Yulong Wu, Viktor Schlegel and Riza Batista-Navarro Department of Computer Science, University of Manchester Manchester, United Kingdom yulong-wu@outlook.com {viktor.schlegel, riza.batista}@manchester.ac.uk Abstract An in-depth analysis of the level of language understanding required by existing Machine Reading Comprehension (MRC) benchmarks can provide insight into the reading capabili- ties of machines. In this paper, we propose an ablation-based methodology to assess the extent to which MRC datasets evaluate the understanding of explicit discourse relations. We define seven MRC skills which require the understanding of different discourse relations. We then introduce ablation methods that ver- ify whether these skills are required to succeed on a dataset. By observing the drop in per- formance of neural MRC models evaluated on the original and the modified dataset, we can measure to what degree the dataset requires these skills, in order to be understood correctly. Experiments on three large-scale datasets with the BERT-base and ALBERT-xxlarge model show that the relative changes for all skills are small (less than 6%). These results imply that most of the answered questions in the ex- amined datasets do not require understanding the discourse structure of the text. To specif- ically probe for natural language understand- ing, there is a need to design more challenging benchmarks that can correctly evaluate the in- tended skills1. 1 Introduction Machine Reading Comprehension (MRC) is con- cerned with the automatic extraction and genera- tion of answers over unstructured textual data. Due to its complexity, the task is seen as suitable for evaluating Natural Language Understanding (NLU) (Chen, 2018). While neural MRC systems achieve impressive performance (Devlin et al., 2019; Lan et al., 2020), it has been revealed by some research efforts that existing MRC benchmarks might be in- sufficient to establish model performance, i.e., that 1Our code is available at https://github.com/ Yulong-W/mrcdr. the models are not being assessed for their capabil- ities to read and comprehend (Jia and Liang, 2017; Mudrakarta et al., 2018; Min et al., 2018; Sugawara et al., 2018; Feng et al., 2018; Jiang and Bansal, 2019; Min et al., 2019; Chen and Durrett, 2019; Schlegel et al., 2020; Sugawara et al., 2020). These analyses provide insights into the weaknesses of modern MRC gold standards. Nonetheless, to stim- ulate the development of robust MRC systems with generalisable NLU capabilities, it is necessary to investigate the strengths and weaknesses of MRC datasets on a deeper level. In the task of MRC, it is assumed that ques- tions test a cognitive process which involves var- ious skills, such as retrieving stored information and performing inferences (Sutcliffe et al., 2013). Therefore, considering metrics that reflect skills required to answer questions is useful for analysing the capabilities of MRC datasets to benchmark NLU (Sugawara et al., 2020). This leads to the following intuition: if a question is …

Content truncated to 500 words