Underspecification in Scene Description-to-Depiction Tasks

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1172–1184 November 20–23, 2022. ©2022 Association for Computational Linguistics 1172 Underspecification in Scene Description-to-Depiction Tasks Ben Hutchinson Google Research, Australia benhutch@google.com Jason Baldridge Google Research, USA jasonbaldridge@google.com Vinodkumar Prabhakaran Google Research, USA vinodkpg@google.com Abstract Questions regarding implicitness, ambiguity and underspecification are crucial for under- standing the task validity and ethical concerns of multimodal image+text systems, yet have received little attention to date. This position paper maps out a conceptual framework to ad- dress this gap, focusing on systems which gen- erate images depicting scenes from scene de- scriptions. In doing so, we account for how texts and images convey meaning differently. We outline a set of core challenges concern- ing textual and visual ambiguity, as well as risks that may be amplified by ambiguous and underspecified elements. We propose and dis- cuss strategies for addressing these challenges, including generating visually ambiguous im- ages, and generating a set of diverse images. 1 Introduction The classic Grounding Problem in AI asks how is it that language can be interpreted as referring to things in the world? It has been argued that demon- strating natural language understanding requires mapping text to something that is non-text and that functions as a model of meaning (e.g., Bender and Koller, 2020). In this view, multimodal models that relate images and language have an important role in pursuing contextualized language understand- ing. Indeed, joint modeling of linguistic and visual signals has been argued to play a critical role in progress towards this ultimate goal, as precursors to modeling relationships between language and the social and physical worlds (Bisk et al., 2020). Recent text-to-image generation systems have demonstrated impressive capabilities (Zhang et al., 2021; Ramesh et al., 2021; Ding et al., 2021; Nichol et al., 2021; Gafni et al., 2022; Ramesh et al., 2022; Saharia et al., 2022; Ramesh et al., 2022; Yu et al., 2022). These employ deep learning methods such as generative adversarial networks (Goodfellow et al., 2014), neural discrete representation learning Figure 1: Generated depictions of the scene “A robot and its pet in a tree.” Many elements are underspecified in the text, e.g., pet type, perspective, and visual style. (van den Oord et al., 2017) combined with auto- regressive models (Brown et al., 2020), and diffu- sion models (Sohl-Dickstein et al., 2015), trained on large datasets of images and aligned texts (Rad- ford et al., 2021; Jia et al., 2021). With such developments in multimodal model- ing and further aspirations towards contextualized language understanding, it is import to better un- derstand both task validity and construct validity in text-to-image systems (Raji et al., 2021). Ethi- cal questions concerning bias, safety and misinfor- mation are increasingly recognized (Saharia et al., 2022; Cho et al., 2022); nevertheless, understand- ing which system behaviors are desirable requires a vocabulary and framework for understanding the diverse and quickly expanding …

Content truncated to 500 words