TopiOCQA: Open-domain Conversational Question Answering with Topic Switching

TopiOCQA: Open-domain Conversational Question Answering with Topic Switching Vaibhav Adlakha1,4 Shehzaad Dhuliawala2 Kaheer Suleman3 Harm de Vries4 Siva Reddy1,5 1Mila, McGill University, Canada 2ETH Z¨urich, Switzerland 3Microsoft Montr´eal, Canada 4ServiceNow Research, Canada 5Facebook CIFAR AI Chair, Canada {vaibhav.adlakha,siva.reddy}@mila.quebec Abstract In a conversational question answering sce- nario, a questioner seeks to extract information about a topic through a series of interdepen- dent questions and answers. As the conver- sation progresses, they may switch to related topics, a phenomenon commonly observed in information-seeking search sessions. However, current datasets for conversational question answering are limiting in two ways: 1) they do not contain topic switches; and 2) they as- sume the reference text for the conversation is given, that is, the setting is not open-domain. We introduce TOPIOCQA (pronounced Tapi- oca), an open-domain conversational dataset with topic switches based on Wikipedia. TOPIOCQA contains 3,920 conversations with information-seeking questions and free-form answers. On average, a conversation in our dataset spans 13 question-answer turns and involves four topics (documents). TOPIOCQA poses a challenging test-bed for models, where efficient retrieval is required on multiple turns of the same conversation, in conjunction with constructing valid responses using conversa- tional history. We evaluate several baselines, by combining state-of-the-art document re- trieval methods with neural reader models. Our best model achieves F1 of 55.8, falling short of human performance by 14.2 points, indicating the difficulty of our dataset. Our dataset and code are available at https:// mcgill-nlp.github.io/topiocqa. 1 Introduction People often engage in information-seeking con- versations to discover new knowledge (Walton, 2019). In such conversations, a questioner (the seeker) asks multiple rounds of questions to an answerer (the expert). As the conversation pro- ceeds, the questioner becomes inquisitive of new but related topics based on the information pro- vided in the answers (Stede and Schlangen, 2004). Such topic switching behaviour is natural in information-seeking conversations and is com- monly observed when people seek information through search engines (Spink et al., 2002). According to Spink et al., people switch from one to ten topics with a mean of 2.11 topic switches per search session. For example, a person can start a search session about tennis, and then land on Roger Federer, and after learning a bit about him may land on his country Switzerland, and spend more time learning about other Swiss athletes. Thanks to tremendous progress in question an- swering research (Rogers et al., 2021), we are coming close to enabling information-seeking conversations with machines (as opposed to just using keywords-based search). In order to real- ize this goal further, it is crucial to construct datasets that contain information-seeking conver- sations with topic switching, and measure progress of conversational models on this task, the two primary contributions of this work. In the literature, a simplified setting of information-seeking conversation known as con- versational question answering (CQA) has been deeply explored (Choi et al., 2018; Reddy et al., 2019). In this task, the entire conversation is based on a given reference text of a topic/entity. While the CQA task is …

Content truncated to 500 words