Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5394–5413 July 9-14, 2023 ©2023 Association for Computational Linguistics Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation Matthieu Futeral1,2 Cordelia Schmid1,2 Ivan Laptev1,2 Benoît Sagot1 Rachel Bawden1 1Inria Paris 2Département d’informatique de l’ENS, CNRS, PSL Research University firstname.lastname@inria.fr Abstract One of the major challenges of machine trans- lation (MT) is ambiguity, which can in some cases be resolved by accompanying context such as images. However, recent work in mul- timodal MT (MMT) has shown that obtaining improvements from images is challenging, lim- ited not only by the difficulty of building ef- fective cross-modal representations, but also by the lack of specific evaluation and training data. We present a new MMT approach based on a strong text-only MT model, which uses neural adapters, a novel guided self-attention mechanism and which is jointly trained on both visually-conditioned masking and MMT. We also introduce CoMMuTE, a Contrastive Multilingual Multimodal Translation Evalu- ation set of ambiguous sentences and their possible translations, accompanied by disam- biguating images corresponding to each transla- tion. Our approach obtains competitive results compared to strong text-only models on stan- dard English→French, English→German and English→Czech benchmarks and outperforms baselines and state-of-the-art MMT systems by a large margin on our contrastive test set. Our code1 and CoMMuTE2 are freely available. 1 Introduction Multimodal machine translation (MMT) typically refers to the use of additional non-textual data in text-based machine translation (MT). Here, we fo- cus on the case where source texts are accompanied by images, the idea being to exploit visual data to improve the translation of ambiguous sentences. For example, in Figure 1, the English word glasses can either be translated as French verres ‘drink- ing vessels’ or lunettes ‘spectacles’, an ambiguity which is resolved using the image. A main research direction of MMT has been how to best exploit image representations and combine 1https://github.com/MatthieuFP/VGAMT 2https://github.com/MatthieuFP/CoMMuTE "The glasses are broken." Translation: "Les verres sont cassés." Translation: "Les lunettes sont cassées." Figure 1: Visual context resolving the ambiguity of English word glasses for English-to-French translation. the image and text modalities (Yin et al., 2020; Caglayan et al., 2021; Calixto et al., 2017; Li et al., 2022). It has typically been difficult to surpass strong text-only baselines, the image modality of- ten being ignored (Wu et al., 2021). A major issue holding back progress is that most current state- of-the-art MMT models (Yin et al., 2020; Elliott and Kádár, 2017; Wu et al., 2021; Li et al., 2022) are trained solely on the ∼30k examples of the Multi30k dataset (Elliott et al., 2016), comprising image captions and their translations. This causes two issues: (i) the models do not exploit the large amount of text-only data available and therefore perform poorly in comparison to state-of-the-art text-only MT systems, and (ii) we show that very few examples require images to be correctly trans- lated, which means that the datasets are ill-adapted to evaluating the use of …

Content truncated to 500 words