Imagination Improves Multimodal Translation

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 130–141, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Imagination Improves Multimodal Translation Desmond Elliott⋆⋄ and ´Akos K´ad´ar† ⋆ILLC, University of Amsterdam ⋄School of Informatics, University of Edinburgh †TiCC, Tilburg University d.elliott@ed.ac.uk, a.kadar@uvt.nl Abstract We decompose multimodal translation into two sub-tasks: learning to translate and learning visually grounded representa- tions. In a multitask learning framework, translations are learned in an attention- based encoder-decoder, and grounded rep- resentations are learned through image representation prediction. Our approach improves translation performance com- pared to the state of the art on the Multi30K dataset. Furthermore, it is equally effective if we train the image pre- diction task on the external MS COCO dataset, and we find improvements if we train the translation model on the external News Commentary parallel text. 1 Introduction Multimodal machine translation is the task of translating sentences in context, such as images paired with a parallel text (Specia et al., 2016). This is an emerging task in the area of multilingual multimodal natural language processing. Progress on this task may prove useful for translating the captions of the images illustrating online news ar- ticles, and for multilingual closed captioning in in- ternational television and cinema. Initial efforts have not convincingly demon- strated that visual context can improve translation quality. In the results of the First Multimodal Translation Shared Task, only three systems out- performed an off-the-shelf text-only phrase-based machine translation model, and the best perform- ing system was equally effective with or without the visual features (Specia et al., 2016). There remains an open question about how translation models should take advantage of visual context. A girl eats a pancake Shared Encoder Attention Average Pool IMAGINET Decoder Image Ein M¨adchen Translation Decoder Figure 1: The Imagination model learns visually- grounded representations by sharing the encoder network between the Translation Decoder with image prediction in the IMAGINET Decoder. We present a multitask learning model that de- composes multimodal translation into learning a translation model and learning visually grounded representations. This decomposition means that our model can be trained over external datasets of parallel text or described images, making it possi- ble to take advantage of existing resources. Fig- ure 1 presents an overview of our model, Imagina- tion, in which source language representations are shared between tasks through the Shared Encoder. The translation decoder is an attention-based neu- ral machine translation model (Bahdanau et al., 2015), and the image prediction decoder is trained to predict a global feature vector of an image that is associated with a sentence (Chrupała et al., 2015, IMAGINET). This decomposition encour- ages grounded learning in the shared encoder be- cause the IMAGINET decoder is trained to imagine 130 the image associated with a sentence. It has been shown that grounded representations are qualita- tively different from their text-only counterparts (K´ad´ar et al., 2016) and correlate better with hu- man similarity judgements (Chrupała et al., 2015). We assess the success of the …

Content truncated to 500 words