A Filtering Approach to Object Region Detection in Multimodal Machine Translation

Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 393–405 September 4–8, 2023, Macau SAR, China. ©2023 The authors. This article is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). 393 A Filtering Approach to Object Region Detection in Multimodal Machine Translation Ali Hatami ali.hatami@insight-centre.org Paul Buitelaar paul.buitelaar@insight-centre.org Mihael Arcan mihael.arcan@insight-centre.org Insight SFI Research Centre for Data Analytics, Data Science Institute, University of Galway, Ireland Abstract Recent studies in Multimodal Machine Translation (MMT) have explored the use of visual in- formation in a multimodal setting to analyze its redundancy with textual information. The aim of this work is to develop a more effective approach to incorporating relevant visual informa- tion into the translation process and improve the overall performance of MMT models. This paper proposes an object-level filtering approach in Multimodal Machine Translation, where the approach is applied to object regions extracted from an image to filter out irrelevant ob- jects based on the image captions to be translated. Using the filtered image helps the model to consider only relevant objects and their relative locations to each other. Different matching methods, including string matching and word embeddings, are employed to identify relevant objects. Gaussian blurring is used to soften irrelevant objects from the image and to evaluate the effect of object filtering on translation quality. The performance of the filtering approaches was evaluated on the Multi30K dataset in English to German, French, and Czech translations, based on BLEU, ChrF2, and TER metrics. 1 Introduction In recent years, neural network-based models have been widely used in translation tasks. Neu- ral Machine Translation (NMT) represents remarkable performance in terms of fluency and precision compared with the previous generations of machine translation (Cho et al., 2014a). Recurrent Neural Network (RNN) with an attention mechanism has found broad application in NMT due to its capability to capture long-term dependencies between the most relevant parts of the source sentence (Cho et al., 2014b). The transformer model has demonstrated remark- able improvements in machine translation tasks. The cross-attention mechanism as a crucial component of the transformer-based model enhances the model’s ability to capture semantic dependencies by combining self-attention, which allows source words to interact with them- selves, with attention mechanisms involving target words (Vaswani et al., 2017). Most current NMT models have shown incredible improvements in the quality of trans- lations, but they rely solely on parallel text corpora for training. However, recent studies (Yao and Wan, 2020; Zhao et al., 2022; Wang and Xiong, 2021) in NMT have increasingly focused on using visual as well as textual content to enhance the quality of translations. Multimodal Machine Translation (MMT), a subarea of NMT, has been introduced to utilise visual informa- tion extracted from other modalities, such as images or videos, to translate an aligned sentence 394 Figure 1: The use of an image helps the translation model disambiguate the word seal in the sentence ”Two boys watch a seal.” and select the correct translation from English to German. in a …

Content truncated to 500 words