Distill The Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2379 - 2390 December 7-11, 2022 ©2022 Association for Computational Linguistics Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation Ru Peng1∗, Yawen Zeng2∗, Junbo Zhao1† 1Zhejiang University, Zhejiang, China 2Tencent WeChat, Shenzhen, China pengru709909347@gmail.com, yawenzeng11@gmail.com, j.zhao@zju.edu.cn Abstract Past works on multimodal machine translation (MMT) elevate bilingual setup by incorpo- rating additional aligned vision information. However, an image-must requirement of the multimodal dataset largely hinders MMT’s development — namely that it demands an aligned form of [image, source text, target text]. This limitation is generally troublesome during the inference phase especially when the aligned image is not provided as in the normal NMT setup. Thus, in this work, we introduce IKD- MMT, a novel MMT framework to support the image-free inference phase via an inversion knowledge distillation scheme. In particular, a multimodal feature generator is executed with a knowledge distillation module, which directly generates the multimodal feature from (only) source texts as the input. While there have been a few prior works entertaining the possibility to support image-free inference for machine translation, their performances have yet to rival the image-must translation. In our experiments, we identify our method as the first image-free approach to comprehensively rival or even surpass (almost) all image-must frameworks, and achieved the state-of-the-art result on the often-used Multi30k benchmark1. 1 Introduction Multimodal machine translation (MMT) is an worthy task of elevating text-only translation by introducing additional image modality (Specia et al., 2016; Elliott et al., 2017; Barrault et al., 2018). Existing works mostly focus on the fusion and alignment of images and texts to improve MMT (Calixto et al., 2017; Ive et al., 2019; Yin et al., 2020), that they have managed to concept-prove the effectiveness of the aligned ∗Both authors contributed equally to this research. †Corresponding author. 1Our code and data are available at: https://github. com/pengr/IKD-mmt/tree/master. (a) Image-must MMT A small boy playing with a wheel. Translation Ein kleiner Junge spielt mit einem Rad. (b) Image-free MMT (Ours) A small boy playing with a wheel. A small boy playing with a wheel. Visual Feature Distillation Textual Feature Training Ein kleiner Junge spielt mit einem Rad. Multi-modal Feature Visual Feature Multi-modal Feature Testing Ein kleiner Junge spielt mit einem Rad. Figure 1: Examples of Image-must MMT (a), and our Image-free MMT (b). During testing, our IKD-MMT does not require the image as input. visual information. Nevertheless, the strict triplet data form of the dataset, in both the training and inference phases, has disabled the MMT model to generalize further. In particular, if we consider using an MMT model to conduct translation for the normal bilingual text translation as in the NMT setup, one must provide the aligned images during inference. And unfortunately, this is not often feasible. This general comparison between image- free and image-must schemes is visually illustrated in Figure 1(a). In hindsight, the quantity and quality of attached images become a bottleneck towards the development of MMT, as …

Content truncated to 500 words