Sheffield MultiMT: Using Object Posterior Predictions for Multimodal Machine Translation

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 470–476 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Sheffield MultiMT: Using Object Posterior Predictions for Multimodal Machine Translation Pranava Madhyastha*, Josiah Wang* and Lucia Specia Department of Computer Science University of Sheffield, UK {p.madhyastha, j.k.wang, l.specia}@sheffield.ac.uk Abstract This paper describes the University of Sheffield’s submission to the WMT17 Multimodal Machine Translation shared task. We participated in Task 1 to de- velop an MT system to translate an im- age description from English to German and French, given its corresponding im- age. Our proposed systems are based on the state-of-the-art Neural Machine Trans- lation approach. We investigate the ef- fect of replacing the commonly-used im- age embeddings with an estimated poste- rior probability prediction for 1,000 object categories in the images. 1 Introduction This paper describes the University of Sheffield’s submission to the second edition of the WMT17 Multimodal Machine Translation shared task. We participate in Task 1, where the challenge is to de- velop a Machine Translation (MT) system to au- tomatically translate image descriptions to a target language, given an image description in a source language and its corresponding image. We sub- mitted systems for translating from English to both German and French. Our submission is based on the state-of-the- art attention-based Neural Machine Translation (NMT) system, which has shown better perfor- mance than conventional phrase-based statistical MT (SMT) systems in the past years. Multimodal NMT systems have been introduced (Elliott et al., 2015; Caglayan et al., 2016; Calixto et al., 2016; Huang et al., 2016) to incorporate visual informa- tion into NMT approaches, most of which condi- tion the NMT on an image representation (typi- *P. Madhyastha and J. Wang contributed equally to this work. cally a vector extracted from a Convolutional Neu- ral Network (CNN) layer). However, it has not been clear thus far whether such image features actually help in the translation task and more im- portant, if so it is not clear which aspects of the image can play a role and how. Recent approaches to Multimodal NMT have used low level image features, including dense fully connected vectors and spatial convolutional representations from an image classification net- work (Elliott et al., 2015; Huang et al., 2016). They also incorporate attention mechanisms (Cal- ixto et al., 2016). However, the effect of image features or the efficacy of the representational con- tribution is still an open research question. For our submission, we propose replacing im- age representations used in current Multimodal NMT systems with a class-based probabilistic dis- tribution that is estimated directly using a state- of-the-art image classification network. The core hypothesis is that such representations offer higher level semantic information and could be more ben- eficial to Multimodal NMT systems. In Section 2 we discuss the motivations be- hind our proposed system. In Section 3 we de- scribe our approach, which uses CNN-based im- age features as input (Section 3.1) to an atten- tion based neural machine …

Content truncated to 500 words