Translation-Enhanced Multilingual Text-to-Image Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9174–9193 July 9-14, 2023 ©2023 Association for Computational Linguistics Translation-Enhanced Multilingual Text-to-Image Generation Yaoyiran Li♠,∗ Ching-Yun Chang♢ Stephen Rawls♢ Ivan Vuli´c♠ Anna Korhonen♠ ♠Language Technology Lab, TAL, University of Cambridge ♢Amazon Alexa AI yl711@cam.ac.uk, {cychang,sterawls}@amazon.com {iv250,alk23}@cam.ac.uk Abstract Research on text-to-image generation (TTI) still predominantly focuses on the English lan- guage due to the lack of annotated image- caption data in other languages; in the long run, this might widen inequitable access to TTI technology. In this work, we thus investi- gate multilingual TTI (termed mTTI) and the current potential of neural machine translation (NMT) to bootstrap mTTI systems. We pro- vide two key contributions. 1) Relying on a multilingual multi-modal encoder, we pro- vide a systematic empirical study of standard methods used in cross-lingual NLP when ap- plied to mTTI: TRANSLATE TRAIN, TRANS- LATE TEST, and ZERO-SHOT TRANSFER. 2) We propose Ensemble Adapter (ENSAD), a novel parameter-efficient approach that learns to weigh and consolidate the multilingual text knowledge within the mTTI framework, mit- igating the language gap and thus improv- ing mTTI performance. Our evaluations on standard mTTI datasets COCO-CN, Multi30K Task2, and LAION-5B demonstrate the poten- tial of translation-enhanced mTTI systems and also validate the benefits of the proposed EN- SAD which derives consistent gains across all datasets. Further investigations on model vari- ants, ablation studies, and qualitative analyses provide additional insights on the inner work- ings of the proposed mTTI approaches. 1 Introduction and Motivation Text-to-Image Generation (TTI) is an emerging yet rapidly growing area, owing its recent progress to ever-growing deep generative models, larger- scale multi-modal datasets, and increasing com- putational resources. The success of recent TTI work is impressive; e.g., it is possible to synthesise not only high-resolution complex scenes (Ramesh et al., 2022; Rombach et al., 2022), but also sur- ∗This work has been done during the author’s internship at Amazon Alexa AI. realist and ‘aesthetics-aware’ paintings (Gallego, 2022). However, current models are made and deployed almost exclusively for the English language (EN). This is primarily due to the lack of annotated image- caption data in other languages, which might result in inequitable access to TTI technology in the long run, especially for low-resource languages (Blasi et al., 2022). Hiring human annotators to write high-quality image descriptions is time-consuming and expensive; ‘gold standard’ data, if it exists at all, is thus typically used for evaluation purposes only (Lan et al., 2017; Aggarwal and Kale, 2020). Even if we put the crucial concerns of data scarcity aside, training state-of-the-art (SotA) TTI models from scratch for each language is techni- cally infeasible and impractical: it would consume massive computational resources, exceeding the capabilities of many research labs (Ramesh et al., 2021; Saharia et al., 2022) and raising concerns of its environmental impact (Schwartz et al., 2020).1 Therefore, in this work, we focus on multilingual TTI (mTTI) through the optics of NLP’s cross- lingual transfer learning methods, leaning on the reasonable …

Content truncated to 500 words