SituatedQA: Incorporating Extra-Linguistic Contexts into QA

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7371–7387 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7371 SITUATEDQA: Incorporating Extra-Linguistic Contexts into QA Michael J.Q. Zhang and Eunsol Choi Computer Science Department The University of Texas at Austin {mjqzhang, eunsol}@utexas.edu Abstract Answers to the same question may change de- pending on the extra-linguistic contexts (when and where the question was asked). To study this challenge, we introduce SITUATEDQA, an open-retrieval QA dataset where systems must produce the correct answer to a question given the temporal or geographical context. To construct SITUATEDQA, we first identify such questions in existing QA datasets. We find that a significant proportion of informa- tion seeking questions have context-dependent answers (e.g. roughly 16.5% of NQ-Open). For such context-dependent questions, we then crowdsource alternative contexts and their cor- responding answers. Our study shows that ex- isting models struggle with producing answers that are frequently updated or from uncommon locations. We further quantify how existing models, which are trained on data collected in the past, fail to generalize to answering ques- tions asked in the present, even when provided with an updated evidence corpus (a roughly 15 point drop in accuracy). Our analysis suggests that open-retrieval QA benchmarks should in- corporate extra-linguistic context to stay rel- evant globally and in the future. Our data, code, and datasheet are available at https: //situatedqa.github.io/. 1 Introduction Language reflects our ever-changing world; thus, the meaning of a sentence depends on a variety of extra-linguistic contexts, including when it was stated, who stated it, and where it was stated. Fig- ure 1 depicts examples of this, where the answer to a question varies based on the temporal and ge- ographical contexts. In this work, we study such questions by exploring the affects of temporal and geographical contexts in open-retrieval QA. Ideally, QA systems should provide up-to-date, localized answers to questions like those in Fig- ure 1 that are situated in the inquirer’s temporal Figure 1: Examples of questions with answers that change depending on the temporal or geographical con- text. and geographical contexts. The existing paradigm for evaluating QA systems, however, makes im- plicit assumptions about time and location and does not measure how well QA models adapt to new contexts. Recent work studying other NLP tasks have shown that this static paradigm misrepresents how models perform in practical settings (Osborne et al., 2014; Wang et al., 2008; Huang and Paul, 2018; Rijhwani and Preotiuc-Pietro, 2020). To ad- dress this challenge, we introduce SITUATEDQA, a dataset comprised of questions from existing open- retrieval QA datasets (Kwiatkowski et al., 2019) that have been re-annotated for their their context- dependence and answers across different situations (i.e., temporal or geographical contexts).1 As we will see in Section 3, a significant pro- 1In this work, we do not model temporal and geographical contexts jointly for simplicity. 7372 Question q Context Type ct Context Value cv Answer a Who composed the music for the first Harry Potter film? - - - …

Content truncated to 500 words