Exploring Better Text Image Translation with Multimodal Codebook

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3479–3491 July 9-14, 2023 ©2023 Association for Computational Linguistics Exploring Better Text Image Translation with Multimodal Codebook Zhibin Lan1,3∗, Jiawei Yu1,3∗, Xiang Li2, Wen Zhang2, Jian Luan2 Bin Wang2, Degen Huang4, Jinsong Su1,3† 1School of Informatics, Xiamen University, China 2Xiaomi AI Lab, Beijing, China 3Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China 4Dalian University of Technology, China {lanzhibin,yujiawei}@stu.xmu.edu.cn jssu@xmu.edu.cn Abstract Text image translation (TIT) aims to translate the source texts embedded in the image to target translations, which has a wide range of appli- cations and thus has important research value. However, current studies on TIT are confronted with two main bottlenecks: 1) this task lacks a publicly available TIT dataset, 2) dominant models are constructed in a cascaded manner, which tends to suffer from the error propagation of optical character recognition (OCR). In this work, we first annotate a Chinese-English TIT dataset named OCRMT30K, providing conve- nience for subsequent studies. Then, we pro- pose a TIT model with a multimodal codebook, which is able to associate the image with rele- vant texts, providing useful supplementary in- formation for translation. Moreover, we present a multi-stage training framework involving text machine translation, image-text alignment, and TIT tasks, which fully exploits additional bilin- gual texts, OCR dataset and our OCRMT30K dataset to train our model. Extensive experi- ments and in-depth analyses strongly demon- strate the effectiveness of our proposed model and training framework.1 1 Introduction In recent years, multimodal machine translation (MMT) has achieved great progress and thus re- ceived increasing attention. Current studies on MMT mainly focus on the text machine transla- tion with scene images (Elliott et al., 2016; Calixto et al., 2017a; Elliott and Kádár, 2017; Libovický et al., 2018; Ive et al., 2019; Zhang et al., 2020; Sulubacak et al., 2020). However, a more common requirement for MMT in real-world applications is text image translation (TIT) (Ma et al., 2022), which aims to translate the source texts embedded in the image to target translations. Due to its wide ∗Equal contribution. †Corresponding author. 1Our code and dataset can be found at https://github. com/DeepLearnXMU/mc_tit Figure 1: An example of text image translation. The Bounding box in red represents the text to be recognized. We can observe that the incorrect OCR result will nega- tively affect the subsequent translation. applications, the industry has developed multiple services to support this task, such as Google Cam- era Translation. Current studies on TIT face two main bottle- necks. First, this task lacks a publicly available TIT dataset. Second, the common practice is to adopt a cascaded translation system, where the texts em- bedded in the input image are firstly recognized by an optical character recognition (OCR) model, and then the recognition results are fed into a text- only neural machine translation (NMT) model for translation. However, such a method tends to …

Content truncated to 500 words