Dealing with Semantic Underspecification in Multimodal NLP

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12098–12112 July 9-14, 2023 ©2023 Association for Computational Linguistics Dealing with Semantic Underspecification in Multimodal NLP Sandro Pezzelle Institute for Logic, Language and Computation University of Amsterdam s.pezzelle@uva.nl Abstract Intelligent systems that aim at mastering lan- guage as humans do must deal with its seman- tic underspecification, namely, the possibility for a linguistic signal to convey only part of the information needed for communication to succeed. Consider the usages of the pronoun they, which can leave the gender and number of its referent(s) underspecified. Semantic un- derspecification is not a bug but a crucial lan- guage feature that boosts its storage and pro- cessing efficiency. Indeed, human speakers can quickly and effortlessly integrate semantically- underspecified linguistic signals with a wide range of non-linguistic information, e.g., the multimodal context, social or cultural conven- tions, and shared knowledge. Standard NLP models have, in principle, no or limited access to such extra information, while multimodal systems grounding language into other modal- ities, such as vision, are naturally equipped to account for this phenomenon. However, we show that they struggle with it, which could negatively affect their performance and lead to harmful consequences when used for applica- tions. In this position paper, we argue that our community should be aware of semantic un- derspecification if it aims to develop language technology that can successfully interact with human users. We discuss some applications where mastering it is crucial and outline a few directions toward achieving this goal. 1 Introduction They put the flowers there. Speakers of a language hear sentences like this every day and have no trou- ble understanding what they mean—and what mes- sage they convey. This is because, in a normal state of affairs, they can count on a wide range of in- formation from the surrounding context, personal knowledge and experience, social or cultural con- ventions, and so on. Upon hearing this sentence, for example, they would know that flowers go into vases, look in the direction where their interlocutor nodded their chin, see a vase with tulips on the win- dowsill, and infer that this is where someone put the flowers. Every time listeners need to count on extra, non-linguistic information to understand a linguis- tic signal, like in this example, it is because the language used is semantically underspecified (Fer- reira, 2008; Frisson, 2009; Harris, 2020b). In the example above, the locative adverb there leaves underspecified a location—where the flowers were put—which would instead be explicitly provided in the semantically more specified sentence They put the flowers in the light blue vase on the win- dowsill at the end of the hallway. According to linguists, indeed, adverbs of place (here, there) are typical examples of semantically underspecified words, as well as adverbs of time (now, today), demonstratives (this, that), quantifiers (few, many), tensed expressions, and some usages of personal pronouns (Lappin, 2000; Harris, 2020b). The reason why semantic underspecification is so widespread …

Content truncated to 500 words