Distilling Translations with Visual Awareness

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6525–6538 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6525 Distilling Translations with Visual Awareness Julia Ive1, Pranava Madhyastha2 and Lucia Specia2 1DCS, University of Sheffield, UK 2Department of Computing, Imperial College London, UK j.ive@sheffield.ac.uk {pranava,l.specia}@imperial.ac.uk Abstract Previous work on multimodal machine transla- tion has shown that visual information is only needed in very specific cases, for example in the presence of ambiguous words where the textual context is not sufficient. As a conse- quence, models tend to learn to ignore this in- formation. We propose a translate-and-refine approach to this problem where images are only used by a second stage decoder. This ap- proach is trained jointly to generate a good first draft translation and to improve over this draft by (i) making better use of the target language textual context (both left and right-side con- texts) and (ii) making use of visual context. This approach leads to the state of the art re- sults. Additionally, we show that it has the ability to recover from erroneous or missing words in the source language. 1 Introduction Multimodal machine translation (MMT) is an area of research that addresses the task of translat- ing texts using context from an additional modal- ity, generally static images. The assumption is that the visual context can help ground the mean- ing of the text and, as a consequence, generate more adequate translations. Current work has fo- cused on datasets of images paired with their de- scriptions, which are crowdsourced in English and then translated into different languages, namely the Multi30K dataset (Elliott et al., 2016). Results from the most recent evaluation cam- paigns in the area (Elliott et al., 2017; Barrault et al., 2018) have shown that visual information can be helpful, as humans generally prefer trans- lations generated by multimodal models than by their text-only counterparts. However, previous work has also shown that images are only needed in very specific cases (Lala et al., 2018). This is also the case for humans. Frank et al. (2018) (see Figure 1) concluded that visual information is needed by humans in the presence of the fol- lowing: incorrect or ambiguous source words and gender-neutral words that need to be marked for gender in the target language. In an experi- ment where human translators were asked to first translate descriptions based on their textual con- text only and then revise their translation based on a corresponding image, they report that these three cases accounted for 62-77% of the revisions in the translations in two subsets of Multi30K. Ambiguities are very frequent in Multi30K, as in most language corpora. Barrault et al. (2018) shows that in its latest test set, 358 (German) and 438 (French) instances (out of 1,000) contain at least one word that has more than one translation in the training set. However, these do not always represent a challenge for translation models: of- ten the text context …

Content truncated to 500 words