Answer Span Correction in Machine Reading Comprehension

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2496–2501 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2496 Answer Span Correction in Machine Reading Comprehension Revanth Gangi Reddy∗, Md Arafat Sultan, Efsun Sarioglu Kayi, Rong Zhang, Vittorio Castelli†, Avirup Sil IBM Research AI g.revanthreddy111@gmail.com, arafat.sultan@ibm.com, {avi,zhangr,vittorio}@us.ibm.com, efsun@gwu.edu Abstract Answer validation in machine reading compre- hension (MRC) consists of verifying an ex- tracted answer against an input context and question pair. Previous work has looked at re-assessing the “answerability” of the ques- tion given the extracted answer. Here we ad- dress a different problem: the tendency of ex- isting MRC systems to produce partially cor- rect answers when presented with answerable questions. We explore the nature of such er- rors and propose a post-processing correction method that yields statistically significant per- formance improvements over state-of-the-art MRC systems in both monolingual and mul- tilingual evaluation. 1 Introduction Extractive machine reading comprehension (MRC) has seen unprecedented progress in recent years (Pan et al., 2019; Liu et al., 2020; Khashabi et al., 2020; Lewis et al., 2019). Nevertheless, existing MRC systems—readers, henceforth—extract only partially correct answers in many cases. At the time of this writing, for example, the top systems on leaderboards like SQuAD (Rajpurkar et al., 2016), HotpotQA (Yang et al., 2018) and Quoref (Dasigi et al., 2019) all have a difference of 5–13 points between their exact match (EM) and F1 scores, which are measures of full and partial overlap with the ground truth answer(s), respectively. Figure 1 shows three examples of such errors that we ob- served in a state-of-the-art (SOTA) RoBERTa-large (Liu et al., 2019) model on the recently released Natural Questions (NQ) (Kwiatkowski et al., 2019) dataset. In this paper, we investigate the nature of such partial match errors in MRC and also their post hoc correction in context. ∗ Work done during AI Residency at IBM Research. † Corresponding author. Q: what type of pasta goes in italian wedding soup GT: usually cavatelli, acini di pepe, pastina, orzo, etc. Prediction: acini di pepe Q: when does precipitate form in a chemical reaction GT: When the reaction occurs in a liquid solution Prediction: Precipitation is the creation of a solid from a solution. When the reaction occurs in a liquid solution Q: what is most likely cause of algal blooms GT: an excess of nutrients, particularly some phosphates Prediction: Freshwater algal blooms are the result of an excess of nutrients Figure 1: Examples of partially correct MRC predic- tions and corresponding ground truth (GT) answers. The reader fails to find a minimal yet sufficient answer in all three cases. Recent work on answer validation (Pe˜nas et al., 2007) has focused on improving the prediction of the answerability of a question given an already extracted answer. Hu et al. (2019) look for support of the extracted answer in local entailments be- tween the answer sentence and the question. Back et al. (2020) propose an attention-based model that explicitly checks if the candidate answer sat- isfies all …

Content truncated to 500 words