Probing the Need for Visual Context in Multimodal Machine Translation

Proceedings of NAACL-HLT 2019, pages 4159–4170 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 4159 Probing the Need for Visual Context in Multimodal Machine Translation Ozan Caglayan LIUM, Le Mans University ozan.caglayan@univ-lemans.fr Pranava Madhyastha Imperial College London pranava@imperial.ac.uk Lucia Specia Imperial College London l.specia@imperial.ac.uk Lo¨ıc Barrault LIUM, Le Mans University loic.barrault@univ-lemans.fr Abstract Current work on multimodal machine trans- lation (MMT) has suggested that the vi- sual modality is either unnecessary or only marginally beneficial. We posit that this is a consequence of the very simple, short and repetitive sentences used in the only available dataset for the task (Multi30K), rendering the source text sufficient as context. In the general case, however, we believe that it is possible to combine visual and textual information in or- der to ground translations. In this paper we probe the contribution of the visual modality to state-of-the-art MMT models by conducting a systematic analysis where we partially de- prive the models from source-side textual con- text. Our results show that under limited tex- tual context, models are capable of leveraging the visual input to generate better translations. This contradicts the current belief that MMT models disregard the visual modality because of either the quality of the image features or the way they are integrated into the model. 1 Introduction Multimodal Machine Translation (MMT) aims at designing better translation systems which take into account auxiliary inputs such as im- ages. Initially organized as a shared task within the First Conference on Machine Translation (WMT16) (Specia et al., 2016), MMT has so far been studied using the Multi30K dataset (Elliott et al., 2016), a multilingual extension of Flickr30K (Young et al., 2014) with translations of the En- glish image descriptions into German, French and Czech (Elliott et al., 2017; Barrault et al., 2018). The three editions of the shared task have seen many exciting approaches that can be broadly cat- egorized as follows: (i) multimodal attention us- ing convolutional features (Caglayan et al., 2016; Calixto et al., 2016; Libovick´y and Helcl, 2017; Helcl et al., 2018) (ii) cross-modal interactions with spatially-unaware global features (Calixto and Liu, 2017; Ma et al., 2017; Caglayan et al., 2017a; Madhyastha et al., 2017) and (iii) the in- tegration of regional features from object detec- tion networks (Huang et al., 2016; Gr¨onroos et al., 2018). Nevertheless, the conclusion about the con- tribution of the visual modality is still unclear: Gr¨onroos et al. (2018) consider their multimodal gains “modest” and attribute the largest gain to the usage of external parallel corpora. Lala et al. (2018) observe that their multimodal word-sense disambiguation approach is not significantly dif- ferent than the monomodal counterpart. The orga- nizers of the latest edition of the shared task con- cluded that the multimodal integration schemes explored so far resulted in marginal changes in terms of automatic metrics and human evaluation (Barrault et al., 2018). In a similar vein, Elliott (2018) demonstrated that MMT models can trans- late without significant performance losses even in the …

Content truncated to 500 words