Resolving Ambiguities in Text-to-Image Generative Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14367–14388 July 9-14, 2023 ©2023 Association for Computational Linguistics Resolving Ambiguities in Text-to-Image Generative Models Ninareh Mehrabi*, Palash Goyal, Apurv Verma, Jwala Dhamala, Varun Kumar, Qian Hu, Kai-Wei Chang, Richard Zemel, Aram Galstyan, Rahul Gupta Amazon Alexa AI-NU Abstract Natural language often contains ambiguities that can lead to misinterpretation and miscom- munication. While humans can handle ambigu- ities effectively by asking clarifying questions and/or relying on contextual cues and common- sense knowledge, resolving ambiguities can be notoriously hard for machines. In this work, we study ambiguities that arise in text-to-image generative models. We curate the Text-to- image Ambiguity Benchmark (TAB) dataset to study different types of ambiguities in text-to- image generative models.1 We then propose the Text-to-ImagE Disambiguation (TIED) frame- work to disambiguate the prompts given to the text-to-image generative models by soliciting clarifications from the end user. Through au- tomatic and human evaluations, we show the effectiveness of our framework in generating more faithful images aligned with end user in- tention in the presence of ambiguities. 1 Introduction Natural conversations contain inherent ambigui- ties due to potentially multiple interpretations of the same utterance. Different types of ambiguities can be attributed to syntax (e.g., “the girl looks at the boy holding a green bag” — is the girl hold- ing the green bag?), semantics (e.g., “a picture of cricket” — is cricket referring to an insect or a game?), and underspecification (e.g., “doctor talk- ing to a nurse” — is the doctor/nurse male or fe- male?). Ambiguities pose an important challenge for many natural language understanding tasks and have been studied extensively in the context of ma- chine translation (Stahlberg and Kumar, 2022), con- versational question answering (Guo et al., 2021), and task-oriented dialogue systems (Qian et al., 2022), among others. In this paper, we study the effect of ambiguity in text-to-image generative models (Ramesh et al., *mninareh@amazon.com 1Data located at: https://github.com/Ninarehm/TAB. Prompt An elephant and a bird flying. Prompt The girl enters the room with flowers. The girl looks at the bird and the butterfly; it is green. Prompt The girl looks at the boy holding a green bag. Prompt Figure 1: Examples of ambiguous prompts and cor- responding generated images. The icon between the images depicts alternative interpretations. The images corresponding to first two prompts are generated by Ope- nAI’s DALL-E (Ramesh et al., 2022), and images corre- sponding to last two prompts are generated by DALL-E Mega (Dayma et al., 2021) models. 2021, 2022; Saharia et al., 2022; Yu et al., 2022) and demonstrate that ambiguous prompts provided to such models might result in undesired outcomes and poor user experience. In particular, ambiguities due to underspecification can lead to biased out- comes with possible implications on fairness of the underlying models (e.g., when prompted with “doc- tor talking to a nurse”, the model might generate images with disproportionate number of male doc- tors and female nurses). We also propose a …

Content truncated to 500 words