PEIT: Bridging the Modality Gap with Pre-trained Models for End-to-End Image Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13433–13447 July 9-14, 2023 ©2023 Association for Computational Linguistics PEIT: Bridging the Modality Gap with Pre-trained Models for End-to-End Image Translation Shaolin Zhu∗, Shangjie Li∗, Yikun Lei, Deyi xiong† College of Intelligence and Computing, Tianjin University, Tianjin, China {zhushaolin, sj_li, yikunlei, dyxiong}@tju.edu.cn Abstract Image translation is a task that translates an image containing text in the source language to the target language. One major challenge with image translation is the modality gap between visual text inputs and textual inputs/outputs of machine translation (MT). In this paper, we propose PEIT, an end-to-end image transla- tion framework that bridges the modality gap with pre-trained models. It is composed of four essential components: a visual encoder, a shared encoder-decoder backbone network, a vision-text representation aligner equipped with the shared encoder and a cross-modal reg- ularizer stacked over the shared decoder. Both the aligner and regularizer aim at reducing the modality gap. To train PEIT, we employ a two- stage pre-training strategy with an auxiliary MT task: (1) pre-training the MT model on the MT training data to initialize the shared encoder-decoder backbone network; and (2) pre-training PEIT with the aligner and regu- larizer on a synthesized dataset with rendered images containing text from the MT training data. In order to facilitate the evaluation of PEIT and promote research on image trans- lation, we create a large-scale image transla- tion corpus ECOIT containing 480K image- translation pairs via crowd-sourcing and man- ual post-editing from real-world images in the e-commerce domain. Experiments on the cu- rated ECOIT benchmark dataset demonstrate that PEIT substantially outperforms both cas- caded image translation systems (OCR+MT) and previous strong end-to-end image transla- tion model, with fewer parameters and faster decoding speed. Codes are available at https: //github.com/lishangjie1/PEIT. 1 Introduction Image translation (IT), transforming an image con- taining text in the source language to an image ∗Equal contribution. †Corresponding author. containing the target translation of the text (Man- simov et al., 2020; Jain et al., 2021), has recently attracted interest (Calixto et al., 2017a; Song et al., 2021). Traditional approaches to IT usually com- bine optical character recognition (OCR) with ma- chine translation (MT) in a cascaded manner, e.g., Google Translate’s Instant Camera1 and Google Lens2. Such pipeline suffers from error propaga- tion and high latency. To address this issue, end- to-end (E2E) image translation, analogous to E2E speech translation that directly translates speech in one language into speech/text in another, has been studied recently (Jain et al., 2021; Mansimov et al., 2020). As a cross-modal task, a major challenge of IT is the representation discrepancy across the textual and visual modality. The text contained in an image is in its visual modality, unlike the text input for text-only machine translation. Its meaning also cor- relates with the visual context in the image. Such visual modality and text-vision correlation make it difficult for IT models to capture the meaning of the text in the …

Content truncated to 500 words