Coreference Reasoning in Machine Reading Comprehension

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5768–5781 August 1–6, 2021. ©2021 Association for Computational Linguistics 5768 Coreference Reasoning in Machine Reading Comprehension Mingzhu Wu1, Nafise Sadat Moosavi1, Dan Roth2, Iryna Gurevych1 1UKP Lab, Technische Universitat Darmstadt 2Department of Computer and Information Science, UPenn 1https://www.ukp.tu-darmstadt.de 2 https://www.seas.upenn.edu/˜danroth/ Abstract Coreference resolution is essential for natu- ral language understanding and has been long studied in NLP. In recent years, as the for- mat of Question Answering (QA) became a standard for machine reading comprehension (MRC), there have been data collection efforts, e.g., Dasigi et al. (2019), that attempt to evalu- ate the ability of MRC models to reason about coreference. However, as we show, coref- erence reasoning in MRC is a greater chal- lenge than earlier thought; MRC datasets do not reflect the natural distribution and, conse- quently, the challenges of coreference reason- ing. Specifically, success on these datasets does not reflect a model’s proficiency in coref- erence reasoning. We propose a methodol- ogy for creating MRC datasets that better re- flect the challenges of coreference reasoning and use it to create a sample evaluation set. The results on our dataset show that state-of- the-art models still struggle with these phe- nomena. Furthermore, we develop an effec- tive way to use naturally occurring corefer- ence phenomena from existing coreference res- olution datasets when training MRC models. This allows us to show an improvement in the coreference reasoning abilities of state-of-the- art models.1 1 Introduction Machine reading comprehension is the ability to read and understand the given passages and answer questions about them. Coreference resolution is the task of finding different expressions that refer to the same real-world entity. The tasks of corefer- ence resolution and machine reading comprehen- sion have moved closer to each other. Converting coreference-related datasets into an MRC format 1The code and the resulting dataset are avail- able at https://github.com/UKPLab/ coref-reasoning-in-qa. improves the performance on some coreference- related datasets (Wu et al., 2020b; Aralikatte et al., 2019). There are also various datasets for the task of reading comprehension on which the model requires to perform coreference reasoning to an- swer some of the questions, e.g., DROP (Dua et al., 2019), DuoRC (Saha et al., 2018), MultiRC (Khashabi et al., 2018), etc. Quoref (Dasigi et al., 2019) is a dataset that is particularly designed for evaluating coreference understanding of MRC models. Figure 1 shows a QA sample from Quoref in which the model needs to resolve the coreference relation between “his” and “John Motteux” to answer the question. Figure 1: A sample from the Quoref dataset. Recent large pre-trained language models reached high performance on Quoref. However, our results and analyses suggest that this dataset contains artifacts and does not reflect the natural distribution and, therefore, the challenges of coref- erence reasoning. As a result, high performances on Quoref do not necessarily reflect the coreference reasoning capabilities of the examined models and answering questions …

Content truncated to 500 words