MMCoQA: Conversational Question Answering over Text, Tables, and Images

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4220 - 4231 May 22-27, 2022 c⃝2022 Association for Computational Linguistics MMCoQA: Conversational Question Answering over Text, Tables, and Images Yongqi Li1, Wenjie Li1, Liqiang Nie2 1The Hong Kong Polytechnic University 2Shandong University {liyongqi0, nieliqiang}@gmail.com cswjli@comp.polyu.edu.hk Abstract The rapid development of conversational assis- tants accelerates the study on conversational question answering (QA). However, the ex- isting conversational QA systems usually an- swer users’ questions with a single knowledge source, e.g., paragraphs or a knowledge graph, but overlook the important visual cues, let alone multiple knowledge sources of differ- ent modalities. In this paper, we hence de- fine a novel research task, i.e., multimodal con- versational question answering (MMCoQA), aiming to answer users’ questions with multi- modal knowledge sources via multi-turn con- versations. This new task brings a series of research challenges, including but not limited to priority, consistency, and complementarity of multimodal knowledge. To facilitate the data-driven approaches in this area, we con- struct the first multimodal conversational QA dataset, named MMConvQA. Questions are fully annotated with not only natural language answers but also the corresponding evidence and valuable decontextualized self-contained questions. Meanwhile, we introduce an end- to-end baseline model, which divides this com- plex research task into question understanding, multi-modal evidence retrieval, and answer ex- traction. Moreover, we report a set of bench- marking results, and the results indicate that there is ample room for improvement. 1 Introduction The ever-increasing variety of information leads to the current information explosion. Question answering (QA) systems play an important role in alleviating information overload by providing users brief and accurate answers. Towards this end, a great many QA systems have been devel- oped by utilizing external knowledge sources to obtain the correct answer, including knowledge- based QA (Deng et al., 2019), document-based QA (Wang et al., 2018), and community-based QA (Fang et al., 2016). Recently, as the rapid Which city features a green copper statue of a woman holding a torch? New York City. What is the largest catholic church in it? St. Patrick's Cathedral. On what dates was the Better with U Tour in the city? March 9, 2012 St. Patrick's Cathedral is the largest Gothic Revival Catholic cathedral in North America Date City March 7, 2012 Nashvile March 9, 2012 New York Images Documents Tables Multimodal Knowledge Collection Answer extraction Ecidence retrieval Figure 1: Illustration of multimodal conversational ques- tion answering. The user asks questions in a conversa- tion and the QA system extracts accurate answers from the multimodal knowledge collection to satisfy users’ information needs. development of conversational assistants, there is growing interest in all matters conversational. Con- versational QA, aiming to satisfy users’ complex information needs via multi-turn conversations, at- tracts a lot of attention. The existing conversational QA systems usu- ally rely on a single knowledge source, e.g., para- graphs or a knowledge graph, and assume it con- tains enough evidence to extract answers to users’ questions. However, these conversational …

Content truncated to 500 words