Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2593–2608 July 9-14, 2023 ©2023 Association for Computational Linguistics CROSS2STRA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment Shengqiong Wu, Hao Fei∗, Wei Ji, Tat-Seng Chua Sea-NExT Joint Lab, School of Computing, National University of Singapore swu@u.nus.edu {haofei37, jiwei, dcscts}@nus.edu.sg, Abstract Unpaired cross-lingual image captioning has long suffered from irrelevancy and disfluency issues, due to the inconsistencies of the seman- tic scene and syntax attributes during transfer. In this work, we propose to address the above problems by incorporating the scene graph (SG) structures and the syntactic constituency (SC) trees. Our captioner contains the semantic structure-guided image-to-pivot captioning and the syntactic structure-guided pivot-to-target translation, two of which are joined via pivot language. We then take the SG and SC struc- tures as pivoting, performing cross-modal se- mantic structure alignment and cross-lingual syntactic structure alignment learning. We further introduce cross-lingual&cross-modal back-translation training to fully align the cap- tioning and translation stages. Experiments on English↔Chinese transfers show that our model shows great superiority in improving captioning relevancy and fluency. 1 Introduction Generating texts to describe images (a.k.a., image captioning) has many real-world applications, such as virtual assistants and image indexing (Fang et al., 2015). Current image captioning models have achieved impressive performance (Jia et al., 2015; Gu et al., 2018a; Ji et al., 2021), yet are mainly limited to the English language due to the large- scale paired image-caption datasets. Subject to the scarcity of paired captioning data, the development of captioning in other languages is thus greatly hin- dered. While manually crafting sufficient paired data is prohibitively expensive, cross-lingual image captioning (Miyazaki and Shimizu, 2016) offers a promising solution, which aims to transfer a cap- tioner trained at resource-rich language (e.g., En- glish) to the resource-scarce language(s) without paired captioning data at target language(s). ∗Corresponding author: Hao Fei A direct approach is to make use of the cur- rent translation techniques, i.e., the pivot language translation method. Here pivot language is the resource-rich language, e.g., English. For example, the pivot-side captioner first generates pivot cap- tions for images, which are then translated into the target-side captions. Or one can create the pseudo image-caption pairs for directly training a target- side captioner, by translating the pivot training cap- tions into the target ones (Lan et al., 2017). How- ever, the above translation-based method suffers from two major issues (cf. 1(a)), including irrele- vancy and disfluency (Song et al., 2019). On the one hand, due to the lack of paired vision contexts, a translated description can easily deviate from the original visual semantics, leading to ambiguous or inaccurate captioning. On the other hand, restricted to the translation system itself, translated texts of- ten suffer from disfluent language, especially for the lengthy and complex descriptions. Some previous efforts are carried out to rectify the above two key errors for better cross-lingual captioning. Lan et al. (2017) solve the transla- tion disfluency issue by estimating the fluency …

Content truncated to 500 words