QAConv: Question Answering on Informative Conversations

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5389 - 5411 May 22-27, 2022 c⃝2022 Association for Computational Linguistics QAConv: Question Answering on Informative Conversations Chien-Sheng Wu1, Andrea Madotto2, Wenhao Liu1, Pascale Fung2, Caiming Xiong1 1Salesforce AI Research 2The Hong Kong University of Science and Technology {wu.jason, wenhao.liu, cxiong}@salesforce.com amadotto@connect.ust.hk, pascale@ece.ust.hk Abstract This paper introduces QAConv, 1, a new ques- tion answering (QA) dataset that uses conver- sations as a knowledge source. We focus on informative conversations, including business emails, panel discussions, and work channels. Unlike open-domain and task-oriented dia- logues, these conversations are usually long, complex, asynchronous, and involve strong domain knowledge. In total, we collect 34,608 QA pairs from 10,259 selected conver- sations with both human-written and machine- generated questions. We use a question gen- erator and a dialogue summarizer as auxil- iary tools to collect and recommend questions. The dataset has two testing scenarios: chunk mode and full mode, depending on whether the grounded partial conversation is provided or retrieved. Experimental results show that state- of-the-art pretrained QA systems have limited zero-shot performance and tend to predict our questions as unanswerable. Our dataset pro- vides a new training and evaluation testbed to facilitate QA on conversations research. 1 Introduction Having conversations is one of the most common ways to share knowledge and exchange informa- tion. Recently, many communication tools and plat- forms are heavily used with the increasing volume of remote working, and how to effectively retrieve information and answer questions based on past conversations becomes more and more important. In this paper, we focus on QA on conversations such as business emails (e.g., Gmail), panel dis- cussions (e.g., Zoom), and work channels (e.g., Slack). Different from daily chit-chat (Li et al., 2017) and task-oriented dialogues (Budzianowski et al., 2018), these conversations are usually long, complex, asynchronous, multi-party, and involve 1Data and code are available at https://github. com/salesforce/QAConv strong domain knowledge. We refer to them as in- formative conversations and an example is shown in Figure 1. However, QA research mainly focuses on docu- ment understanding (e.g., Wikipedia) not dialogue understanding, and dialogues have significant dif- ferences with documents in terms of data format and wording style, and important information is scattered in multiple speakers and turns (Wolf et al., 2019b; Wu et al., 2020). Moreover, existing work related to QA and conversational AI focuses on conversational QA (Reddy et al., 2019; Choi et al., 2018) instead of QA on conversations. Conversa- tional QA has sequential dialogue-like QA pairs that are grounded on a short document paragraph, but what we are more interested in is to have QA pairs grounded on conversations, treating past dia- logues as a knowledge source. QA on conversation has several unique chal- lenges: 1) information is distributed across mul- tiple speakers and scattered among dialogue turns; 2) Harder coreference resolution problem of speak- ers and entities, and 3) missing supervision as no training data in such format is available. The most related work …

Content truncated to 500 words