CREPE: Open-Domain Question Answering with False Presuppositions

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10457–10480 July 9-14, 2023 ©2023 Association for Computational Linguistics CREPE: Open-Domain Question Answering with False Presuppositions Xinyan Velocity Yu† Sewon Min† Luke Zettlemoyer† Hannaneh Hajishirzi†,‡ †University of Washington ‡Allen Institute for Artificial Intelligence {xyu530,sewon,lsz,hannaneh}@cs.washington.edu Abstract When asking about unfamiliar topics, informa- tion seeking users often pose questions with false presuppositions. Most existing question answering (QA) datasets, in contrast, assume all questions have well defined answers. We introduce CREPE, a QA dataset containing a natural distribution of presupposition failures from online information-seeking forums. We find that 25% of questions contain false pre- suppositions, and provide annotations for these presuppositions and their corrections. Through extensive baseline experiments, we show that adaptations of existing open-domain QA mod- els can find presuppositions moderately well, but struggle when predicting whether a presup- position is factually correct. This is in large part due to difficulty in retrieving relevant evidence passages from a large text corpus. CREPE pro- vides a benchmark to study question answering in the wild, and our analyses provide avenues for future work in better modeling and further studying the task.1 1 Introduction When an information-seeking user poses a ques- tion about the topic they are unfamiliar with, they can often introduce false presuppositions (Kaplan, 1978; Duží and ˇCíhalová, 2015) which are assumed but not directly stated. For instance, the ques- tion in Figure 1 incorrectly presupposes that the equal and opposite reactions in Newton’s law ap- ply to the same object. Although such a question is unanswerable, we might still hope to identify the confusion and explain it to the user. This functionality goes well beyond prior open-domain QA task formulations, which focus on questions with a valid direct answer (Rajpurkar et al., 2016; Kwiatkowski et al., 2019) or that are unanswer- able from lack of evidence (Rajpurkar et al., 2018; Choi et al., 2018; Asai and Choi, 2021). While 1The data, baselines, and the evaluation script are available at github.com/velocityCavalry/CREPE. Question: If there's an equal and opposite reaction for everything, how does any action happen? Isn't it balanced out by the opposite reaction? False presupposition: The equal and opposite reaction applies to the same object. Correction: Based on Newton's Law of Motion, the equal and opposite reaction applies to the other object. Only forces that are applied to the same object would be cancelled out. Newton’s laws of motion From Wikipedia, the free encyclopedia Overly brief paraphrases of the third law, like "action equals reaction" might have caused confusion among generations of students: the "action" and "reaction" apply to different bodies. Net force From Wikipedia, the free encyclopedia In mechanics, the net force is the vector sum of forces acting on a particle or object. The net force is a single force that replaces the effect of the original forces on the particle's motion. Reaction (Physics) From Wikipedia, the free encyclopedia … One problem frequently observed by physics educators is that students tend to apply …

Content truncated to 500 words