Multilingual Conceptual Coverage in Text-to-Image Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4831–4848 July 9-14, 2023 ©2023 Association for Computational Linguistics Multilingual Conceptual Coverage in Text-to-Image Models Michael Saxon University of California, Santa Barbara saxon@ucsb.edu William Yang Wang University of California, Santa Barbara william@cs.ucsb.edu EN ES DE ZH JP HE ID EN ES DE ZH JP HE ID EN ES DE ZH JP HE ID concept: dog concept: airplane concept: face AltDiffusion DallE 2 StableDiffusion 2 DallE mega 1 Figure 1: A selection of images generated by DALLE-mega, Stable Diffusion 2, DALLE-2, and AltDiffusion, illustrating their conceptual coverage of “dog,” “airplane,” and “face” across English, Spanish, German, Chinese (simplified), Japanese, Hebrew, and Indonesian. Coverage of the concepts varies considerably across model and language, and can be observed in the consistency and correctness of images generated under simple prompts. Abstract We propose “Conceptual Coverage Across Languages” (CoCo-CroLa), a technique for benchmarking the degree to which any gener- ative text-to-image system provides multilin- gual parity to its training language in terms of tangible nouns. For each model we can as- sess “conceptual coverage” of a given target language relative to a source language by com- paring the population of images generated for a series of tangible nouns in the source language to the population of images generated for each noun under translation in the target language. This technique allows us to estimate how well- suited a model is to a target language as well as identify model-specific weaknesses, spuri- ous correlations, and biases without a-priori assumptions. We demonstrate how it can be used to benchmark T2I models in terms of mul- tilinguality, and how despite its simplicity it is a good proxy for impressive generalization. 1 Introduction Neural text-to-image models convert text prompts into images (Mansimov et al., 2016; Reed et al., 2016) using internal representations reflective of the training data population. Advancements in con- ditional language modeling (Lewis et al., 2020), variational autoencoders (Kingma and Welling, 2014), GANs (Goodfellow et al., 2020), multi- modal representations (Radford et al., 2021), and latent diffusion models (Rombach et al., 2022) have given rise to sophisticated text-to-image (T2I) sys- tems that exhibit impressive semantic generaliza- tion capabilities, with which they generate coher- ent, visually-appealing images with novel combi- nations of objects, scenarios, and styles (Ramesh et al., 2021). Their semantic latent spaces (Kwon et al., 2022) ground words to associated visuals (Hutchinson et al., 2022). Characterizing the limits of these systems’ capabilities is a challenge. They 4831 Figure 2: We hypothesize that a model’s ability to gen- erate creative, compositional images depicting tangible concepts (e.g., astronaut, horse, soup, bear) is predicated on its ability to generate simple images of the concepts alone. Samples from Ramesh et al. (2022). are composed of elements trained on incomprehen- sibly large (Prabhu and Birhane, 2020; Jia et al., 2021), web-scale data (Gao et al., 2021; Schuh- mann et al., 2021), hindering training-data-centric model analysis (Mitchell et al., 2019; Gebru et al., 2021) to address …

Content truncated to 500 words