MeetingQA: Extractive Question-Answering on Meeting Transcripts

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15000–15025 July 9-14, 2023 ©2023 Association for Computational Linguistics MEETINGQA: Extractive Question-Answering on Meeting Transcripts Archiki Prasad1 Trung Bui2 Seunghyun Yoon2 Hanieh Deilamsalehy2 Franck Dernoncourt2 Mohit Bansal1 1UNC Chapel Hill 2Adobe Research {archiki, mbansal}@cs.unc.edu {bui, syoon, deilamsa, dernonco}@adobe.com Abstract With the ubiquitous use of online meeting plat- forms and robust automatic speech recognition systems, meeting transcripts have emerged as a promising domain for natural language tasks. Most recent works on meeting transcripts pri- marily focus on summarization and extraction of action items. However, meeting discussions also have a useful question-answering (QA) component, crucial to understanding the dis- course or meeting content, and can be used to build interactive interfaces on top of long tran- scripts. Hence, in this work, we leverage this inherent QA component of meeting discussions and introduce MEETINGQA, an extractive QA dataset comprising of questions asked by meet- ing participants and corresponding responses. As a result, questions can be open-ended and actively seek discussions, while the answers can be multi-span and distributed across mul- tiple speakers. Our comprehensive empiri- cal study of several robust baselines includ- ing long-context language models and recent instruction-tuned models reveals that models perform poorly on this task (F1 = 57.3) and severely lag behind human performance (F1 = 84.6), thus presenting a challenging new task for the community to improve upon.1 1 Introduction Millions of meetings occur every day worldwide, which results in vast amounts of meeting tran- scripts. Meeting transcripts are typically long doc- uments, often domain-specific depending on the subject matter, and contain a lot of information. Basic tasks such as catching up with a missed meet- ing, looking up a specific discussion or response to a query can be time-consuming. These tasks can be facilitated by NLP systems, including summariza- tion and question-answering. To this end, several publicly available small-scale corpora of meeting 1MEETINGQA data and code is available at https:// archiki.github.io/meetingqa.html Figure 1: Representative example from meeting tran- script segment in MEETINGQA. The question and anno- tated answer are highlighted in red and blue respectively. transcripts have been released (Carletta et al., 2005; Janin et al., 2003; Garofolo et al., 2004, inter alia). Prior NLP work on meeting transcripts mainly focuses on summarization (Oya et al., 2014; Li et al., 2019; Zhu et al., 2020, inter alia). However, lack of annotated data impedes research on other important NLP tasks in this domain. To address this gap, we introduce a question-answering (QA) task based on conversations in meeting transcripts. Specifically, we consider questions asked by par- ticipants during the meeting and aim to extract cor- responding answer spans from relevant discussions among meeting participants (refer to Figure 1). This task has several practical applications such as building an interactive meeting browser/interface for navigating through transcripts and informing tasks such as meeting summarization and handling action items involving QA pairs (Kathol and Tur, 2008; August et al., 2022). 15000 While standard QA datasets …

Content truncated to 500 words