MSCTD: A Multimodal Sentiment Chat Translation Dataset

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2601 - 2613 May 22-27, 2022 c⃝2022 Association for Computational Linguistics MSCTD: A Multimodal Sentiment Chat Translation Dataset Yunlong Liang1∗, Fandong Meng2∗, Jinan Xu1†, Yufeng Chen1 and Jie Zhou2 1Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University, Beijing, China 2Pattern Recognition Center, WeChat AI, Tencent Inc, China {yunlongliang,jaxu,chenyf}@bjtu.edu.cn {fandongmeng,withtomzhou}@tencent.com Abstract Multimodal machine translation and textual chat translation have received considerable at- tention in recent years. Although the conversa- tion in its natural form is usually multimodal, there still lacks work on multimodal machine translation in conversations. In this work, we introduce a new task named Multimodal Chat Translation (MCT), aiming to generate more accurate translations with the help of the as- sociated dialogue history and visual context. To this end, we firstly construct a Multimodal Sentiment Chat Translation Dataset (MSCTD) containing 142,871 English-Chinese utterance pairs in 14,762 bilingual dialogues and 30,370 English-German utterance pairs in 3,079 bilin- gual dialogues. Each utterance pair, corre- sponding to the visual context that reflects the current conversational scene, is annotated with a sentiment label. Then, we benchmark the task by establishing multiple baseline systems that incorporate multimodal and sentiment fea- tures for MCT. Preliminary experiments on four language directions (English↔Chinese and English↔German) verify the potential of contextual and multimodal information fusion and the positive impact of sentiment on the MCT task. Additionally, as a by-product of the MSCTD, it also provides two new bench- marks on multimodal dialogue sentiment anal- ysis. Our work can facilitate research on both multimodal chat translation and multimodal di- alogue sentiment analysis.1 1 Introduction Multimodal machine translation (Huang et al., 2016; Calixto and Liu, 2017) and textual chat trans- lation (Wang et al., 2016; Farajian et al., 2020; Liang et al., 2021a) mainly focus on investigating ∗Equal contribution. Work was done when Yunlong were interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China. †Jinan Xu is the corresponding author. 1The code, data, and image features are publicly available at: https://github.com/XL2248/MSCTD the potential visual features and dialogue context, respectively. Both of them have received much attention. Although plenty of studies on them have been carried out based on either image cap- tions (Calixto et al., 2017, 2019; Ive et al., 2019; Yin et al., 2020; Yao and Wan, 2020) or textual dialogues (Wang et al., 2017; Maruf et al., 2018; Liang et al., 2021c), to our knowledge, little re- search work has been devoted to multimodal ma- chine translation in conversations. One important reason is the lack of multimodal bilingual conver- sational datasets. Generally, conversation in its natural form is multimodal (Poria et al., 2019; Liang et al., 2021b). When humans converse, what a speaker would say next depends largely on what he/she sees. That is, the visual information plays a key role in (i) supple- menting some crucial scene information (e.g., the specific locations or objects, or facial expressions), (ii) resolving ambiguous multi-sense words (e.g., bank), and …

Content truncated to 500 words