Neural Machine Translation with Phrase-Level Universal Visual Representations

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5687 - 5698 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Neural Machine Translation with Phrase-Level Universal Visual Representations Qingkai Fang1,2, Yang Feng1,2∗ 1 Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2 University of Chinese Academy of Sciences, Beijing, China {fangqingkai21b,fengyang}@ict.ac.cn Abstract Multimodal machine translation (MMT) aims to improve neural machine translation (NMT) with additional visual information, but most existing MMT methods require paired input of source sentence and image, which makes them suffer from shortage of sentence-image pairs. In this paper, we propose a phrase-level retrieval-based method for MMT to get visual information for the source input from exist- ing sentence-image data sets so that MMT can break the limitation of paired sentence-image input. Our method performs retrieval at the phrase level and hence learns visual informa- tion from pairs of source phrase and grounded region, which can mitigate data sparsity. Fur- thermore, our method employs the conditional variational auto-encoder to learn visual repre- sentations which can filter redundant visual in- formation and only retain visual information related to the phrase. Experiments show that the proposed method significantly outperforms strong baselines on multiple MMT datasets, es- pecially when the textual context is limited. 1 Introduction Multimodal machine translation (MMT) introduces visual information into neural machine transla- tion (NMT), which assumes that additional visual modality could improve NMT by grounding the language into a visual space (Lee et al., 2018). However, most existing MMT methods require additional input of images to provide visual rep- resentations, which should match with the source sentence. Unfortunately, in practice it is difficult to get this kind of pairwise input of text and images which hinders the applications of MMT. What is worse, to train an MMT model, the training data still involves the target sentence besides the source ∗Corresponding author: Yang Feng. Code is publicly available at https://github.com/ ictnlp/PLUVR. sentence and the image, which is costly to collect. As a result, the MMT model is usually trained on a small Multi30K (Elliott et al., 2016) data set, which limits the performance of MMT. Therefore, it is necessary to utilize the separate image data set to obtain visual representations to break the constraints of pairwise input. Towards this end, some researchers (Zhang et al., 2020; Wu et al., 2021) propose to integrate a re- trieval module into NMT, which retrieve images re- lated to the source sentence from existing sentence- image pairs as complementary input, and then use a pre-trained convolutional neural network (CNN) to encode the images. However, such sentence-level retrieval usually suffers from sparsity as it is dif- ficult to get the images that properly match with the source sentence. Besides, visual features out- putted by the CNN contain richer information (e.g., color, size, shape, texture, and background) than the source text, thus encoding them in a bundle without any filtering will introduce noise into the model. To …

Content truncated to 500 words