Answer Uncertainty and Unanswerability in Multiple-Choice Machine Reading Comprehension

Findings of the Association for Computational Linguistics: ACL 2022, pages 1020 - 1034 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Answer Uncertainty and Unanswerability in Multiple-Choice Machine Reading Comprehension Vatsal Raina Cambridge University vr311@cam.ac.uk Mark Gales Cambridge University mjfg@cam.ac.uk Abstract Machine reading comprehension (MRC) has drawn a lot of attention as an approach for as- sessing the ability of systems to understand natural language. Usually systems focus on se- lecting the correct answer to a question given a contextual paragraph. However, for many appli- cations of multiple-choice MRC systems there are two additional considerations. For multiple- choice exams there is often a negative marking scheme; there is a penalty for an incorrect an- swer. This means that the system is required to have an idea of the uncertainty in the pre- dicted answer. The second consideration is that many multiple-choice questions have the op- tion of none of the above (NOA) indicating that none of the answers is applicable, rather than there always being the correct answer in the list of choices. This paper investigates both of these issues by making use of predictive uncer- tainty. It is shown that uncertainty does allow questions that the system is not confident about to be detected. Additionally we show that un- certainty outperforms a system explicitly built with an NOA option for the ReClor corpus. 1 Introduction Machine reading comprehension (MRC), where the correct answer must be deduced for a question from a context paragraph, plays a crucial role in de- veloping systems for natural language processing and understanding. In recent years, popular MRC datasets (Richardson et al., 2013; Chen et al., 2016; Lai et al., 2017; Trischler et al., 2017; Rajpurkar et al., 2018; Yang et al., 2018; Yu et al., 2020) have consistently observed increasingly competitive sys- tems topping public leaderboards (Trischler et al., 2016; Dhingra et al., 2017; Zhang et al., 2021; Ya- mada et al., 2020; Zaheer et al., 2020; Wang et al., 2021) and surpassing human performance. How- ever, systems in deployment should not necessarily always aim to answer a posed reading comprehen- sion question. There are two modes of interest in which an MRC system may choose to abstain from giving an answer: answer uncertainty and unanswerability. If a system is uncertain about its prediction, it is likely that the predicted answer will be incorrect. In particular, negative marking schemes, which are shown to improve the reliabil- ity of multiple-choice assessment as guessing is deterred (Holt, 2006), penalise a system for predict- ing an incorrect answer while abstaining carries no penalty, and of course the correct answer has a pos- itive reward. In such cases, it would be sensible for a system to abstain from answering if there is an- swer uncertainty in the prediction. Unanswerability is where the answer to a question is not deducible from the associated context. Consequently, a sys- tem should abstain from answering a question if it believes the answer is not present in the context. Answer uncertainty is when …

Content truncated to 500 words