McQueen: a Benchmark for Multimodal Conversational Query Rewrite

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4834 - 4844 December 7-11, 2022 ©2022 Association for Computational Linguistics McQueen: a Benchmark for Multimodal Conversational Query Rewrite Yifei Yuan1∗, Chen Shi2, Runze Wang2, Liyi Chen3, Feijun Jiang2, Yuan You2, and Wai Lam1 1The Chinese University of Hong Kong 2Alibaba Group 3Nankai University {yfyuan,wlam}@se.cuhk.edu.hk {deling.sc,yunze.wrz,feijun.jiangfj,youyuan.yy}@alibaba-inc.com liyichen@mail.nankai.edu.cn Abstract The task of query rewrite aims to convert an in-context query to its fully-specified version where ellipsis and coreference are completed and referred-back according to the history con- text. Although much progress has been made, less efforts have been paid to real scenario conversations that involve drawing informa- tion from more than one modalities. In this paper, we propose the task of multimodal con- versational query rewrite (McQR), which per- forms query rewrite under the multimodal vi- sual conversation setting. We collect a large- scale dataset named McQueen based on manual annotation, which contains 15k visual conver- sations and over 80k queries where each one is associated with a fully-specified rewrite ver- sion. In addition, for entities appearing in the rewrite, we provide the corresponding image box annotation. We then use the McQueen dataset to benchmark a state-of-the-art method for effectively tackling the McQR task, which is based on a multimodal pre-trained model with pointer generator. Extensive experiments are performed to demonstrate the effectiveness of our model on this task1. 1 Introduction Recent years have witnessed an increasing attention in conversational-related tasks, such as conversa- tional question answering (Choi et al., 2018; Reddy et al., 2019), visual conversation modeling (Das et al., 2017), etc. One main challenge in multi-turn conversation modeling is that information from context history is easy to be abbreviated or omit- ted in the follow-up queries, causing the so-called coreference and ellipsis. To address this concern, ∗Work done when Yifei Yuan was an intern at Alibaba. This work was supported by Alibaba Group through Alibaba Research Intern Program, and a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Codes: 14200620). 1The dataset and code of this paper are both available in https://github.com/yfyuan01/MQR !!: Is it a black Labrador? !! ∗: Is the dog a black Labrador? #!: Yes. !#: How may people are there in the scene? !# ∗: How may people are there in the scene? ##: Just one. !$: Can you see other people? !$∗: Can you see other people except for the man? #$: No. the dog the man Figure 1: An example of the mulitmodal query rewrite task, where Q, Q∗, and A denote the queries, their cor- responding rewrites and the answers. Red color denotes the coreference rewrite part and blue denotes the ellipsis rewrite part. Image boxes are utilized for representing the area of the entities in the rewrite. the task of query rewrite (Elgohary et al., 2019; Pan et al., 2019; Su et al., 2019) aims to reconstruct the original query to a fully specified form based on its history context. The …

Content truncated to 500 words