VISA: An Ambiguous Subtitles Dataset for Visual Scene-aware Machine Translation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6735–6743 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6735 VISA: An Ambiguous Subtitles Dataset for Visual Scene-Aware Machine Translation Yihang Li, Shuichiro Shimizu, Weiqi Gu, Chenhui Chu, Sadao Kurohashi Kyoto University, Kyoto, Japan {liyh, sshimizu, gu, chu, kuro}@nlp.ist.i.kyoto-u.ac.jp Abstract Existing multimodal machine translation (MMT) datasets consist of images and video captions or general subtitles, which rarely contain linguistic ambiguity, making visual information not so effective to generate appropriate translations. We introduce VISA, a new dataset that consists of 40k Japanese–English parallel sentence pairs and corresponding video clips with the following key features: (1) the parallel sentences are subtitles from movies and TV episodes; (2) the source subtitles are ambiguous, which means they have multiple possible translations with different meanings; (3) we divide the dataset into Polysemy and Omission according to the cause of ambiguity. We show that VISA is challenging for the latest MMT system, and we hope that the dataset can facilitate MMT research. The VISA dataset is available at: https://github.com/ku-nlp/VISA. Keywords: multimodality, machine translation, ambiguity, scene-awareness 1. Introduction Neural machine translation (NMT) models relying on text data have achieved state-of-the-art performance for domains with little ambiguity in data. However, in sev- eral other domains, particularly real-time domains such as spoken language or sports commentary, the word sense ambiguity of verbs and nouns has a significant impact on translation quality (Gu et al., 2021). Mul- timodal machine translation (MMT) focuses on using visual data as auxiliary information to tackle the ambi- guity problem. The contextual information in the visual data assists in reducing the ambiguity of nouns or verbs in the source text data. Previous MMT studies mainly focus on the image- guided machine translation (IMT) task (Elliott et al., 2016; Zhao et al., 2020), where, given an image and a source sentence, the goal is to enhance the quality of translation by leveraging their semantic correspon- dence to the image. Resolving ambiguities through vi- sual cues is one of the main motivations behind this task. However, videos are better information sources than images because one video contains an ordered se- quence of frames and provides much more visual fea- tures such as motion features. Recently, some stud- ies have started to focus on the video-guided machine translation (VMT) task (Wang et al., 2019; Hirasawa et al., 2020). VMT datasets face the problem of data scarcity. How2 (Sanabria et al., 2018) and VATEX (Wang et al., 2019) datasets are recent efforts to alleviate the problem. In addition, previous datasets are limited to general sub- titles or video captions which are actually descriptions of video clips. Besides the lack of practical use of cap- tion MT, it has been shown that general subtitles and caption MT essentially do not require visual informa- tion due to the lack of language ambiguity in captions (Caglayan et al., 2019). To this end, we construct a new large-scale parallel sub- titles dataset1 for VMT research, …

Content truncated to 500 words