Generative Imagination Elevates Machine Translation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5738–5748 June 6–11, 2021. ©2021 Association for Computational Linguistics 5738 Generative Imagination Elevates Machine Translation Quanyu Long1, Mingxuan Wang2, and Lei Li2 1Nanyang Technological University, Singapore 2ByteDance AI Lab, China quanyu001@e.ntu.edu.sg; {wangmingxuan.89, lileilab}@bytedance.com Abstract There are common semantics shared across text and images. Given a sentence in a source language, whether depicting the visual scene helps translation into a target language? Ex- isting multimodal neural machine translation methods (MNMT) require triplets of bilingual sentence - image for training and tuples of source sentence - image for inference. In this paper, we propose ImagiT, a novel ma- chine translation method via visual imagina- tion. ImagiT first learns to generate visual rep- resentation from the source sentence, and then utilizes both source sentence and the “imag- ined representation” to produce a target trans- lation. Unlike previous methods, it only needs the source sentence at the inference time. Ex- periments demonstrate that ImagiT benefits from visual imagination and significantly out- performs the text-only neural machine trans- lation baselines. Further analysis reveals that the imagination process in ImagiT helps fill in missing information when performing the degradation strategy. 1 Introduction Visual foundation has been introduced in a novel multimodal Neural Machine Translation (MNMT) task (Specia et al., 2016; Elliott et al., 2017; Bar- rault et al., 2018), which uses bilingual (or multi- lingual) parallel corpora annotated by images de- scribing sentences’ contents (see Figure 1(a)). The superiority of MNMT lies in its ability to use visual information to improve the quality of translation, but its effectiveness largely depends on the avail- ability of data sets, especially the quantity and qual- ity of annotated images. In addition, because the cost of manual image annotation is relatively high, at this stage, MNMT is mostly applied on a small and specific dataset, Multi30K (Elliott et al., 2016), and is not suitable for large-scale text-only Neu- ral Machine Translation (NMT) (Bahdanau et al., Model A bird flies over the water Ein Vogel fliegt über das Wasser (a) Multimodal NMT Model Visual representation A bird flies over the water Ein Vogel fliegt über das Wasser “Imagination” (b) ImagiT Figure 1: The problem setup of our proposed ImagiT is different from existing multimodal NMT. A multi- modal NMT model takes both text and paired image as the input, while ImagiT takes only sentence in the source language as the usual NMT task. ImagiT syn- thesizes an image and utilize the internal visual repre- sentation to assist translation. 2015; Vaswani et al., 2017). Such limitations hin- der the applicability of visual information in NMT. To address the bottlenecks mentioned above, Zhang et al. (2020) propose to build a lookup table from an image dataset and then using the search- based method to retrieve pictures that match the source language keywords. However, the lookup table is built from Multi30K, which leads to a rel- atively limited coverage of the pictures, and po- …

Content truncated to 500 words