Multimodal Transformer for Multimodal Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4346–4350 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4346 Multimodal Transformer for Multimodal Machine Translation Shaowei Yao, Xiaojun Wan Center for Data Science, Peking University Wangxuan Institute of Computer Technology, Peking University The MOE Key Laboratory of Computational Linguistics, Peking University {yaosw,wanxiaojun}@pku.edu.cn Abstract Multimodal Machine Translation (MMT) aims to introduce information from other modality, generally static images, to improve the transla- tion quality. Previous works propose various incorporation methods, but most of them do not consider the relative importance of multi- ple modalities. In MMT, equally treating text and images may encode too much irrelevant in- formation from images which may introduce noise. In this paper, we propose the multi- modal self-attention in Transformer to solve the issues above. The proposed method learns the representations of images based on the text, which avoids encoding irrelevant informa- tion in images. Experiments and visualization analysis demonstrate that our model benefits from visual information and substantially out- performs previous works and competitive base- lines in terms of various metrics. 1 Introduction Multimodal machine translation (MMT) is a novel machine translation (MT) task which aims at de- signing better translation systems using context from an additional modality, usually images (See Figure 1). It initially organized as a shared task within the First Conference on Machine Transla- tion (Specia et al., 2016; Elliott et al., 2017; Bar- rault et al., 2018). Current works focus on the dataset named Multi30k (Elliott et al., 2016), a multilingual extension of Flickr30k dataset with translations of the English image descriptions into different languages. Previous works propose various incorporation methods. Calixto and Liu (2017) utilize global im- age features to initialize the encoder/decoder hid- den states of RNN. Elliott and K´ad´ar (2017) model the source sentence and reconstruct the image repre- sentation jointly via multi-task learning. Recently, Ive et al. (2019) propose a translate-and-refine ap- Figure 1: An Example for Multimodal Machine Trans- lation. proach using two-stage decoder based on Trans- former (Vaswani et al., 2017). Calixto et al. (2019) put forward a latent variable model to learn the in- teraction between visual and textual features. How- ever, in multimodal tasks the different modalities usually are not equally important. For example, in MMT the text is obviously more important than images. Although the image carries richer infor- mation, it also contains more irrelevant content. If we directly encode the image features, it may introduce a lot of noise. To address the issues above, we propose the mul- timodal Transformer. The proposed model does not directly encode image features. Instead, the hidden representations of images are induced from the text under the guide of image-aware attention. Mean- while, we introduce a better way to incorporate information from other modality based on a graph perspective of Transformer. Experimental results and visualization show that our model can make good use of visual information and substantially outperforms the current state of the art. 2 Methodology Our model is …

Content truncated to 500 words