A Formal Analysis of Multimodal Referring Strategies Under Common Ground

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5919–5927 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5919 A Formal Analysis of Multimodal Referring Strategies Under Common Ground Nikhil Krishnaswamy and James Pustejovsky Brandeis University Department of Computer Science Walthan, MA, USA {nkrishna,jamesp}@brandeis.edu Abstract In this paper, we present an analysis of computationally generated mixed-modality definite referring expressions using combinations of gesture and linguistic descriptions. In doing so, we expose some striking formal semantic properties of the interactions between gesture and language, conditioned on the introduction of content into the common ground between the (computational) speaker and (human) viewer, and demonstrate how these formal features can contribute to training better models to predict viewer judgment of referring expressions, and potentially to the generation of more natural and informative referring expressions. Keywords: multimodality, interfaces, referring expressions, semantics, common ground 1. Introduction Multimodality has been a topic of study in computational linguistics and natural language processing since at least the mid-1990s (Johnston et al., 1997), but has seen in- creased interest from the CL/NLP communities in recent years. This has been due to a number of factors, including the increase in processing power; the availability of large datasets of text, images, and video; the rise of depth sensors (e.g., Microsoft Kinect); and the availability of GPUs for deep model training. This has resulted in a number of new datasets and approaches to cross-modal linking (Yatskar et al., 2016; Goyal et al., 2017), shared tasks (Barrault et al., 2018), and grounding tasks (Beinborn et al., 2018; Zhou et al., 2018). The most common modalities under study in the CL/NLP communities are text, audio/speech, and images/video, but “modality” can in principle refer to any channel of infor- mation. Therefore, multi-channel transmission of informa- tion can be separated by channel into the particular infor- mation transmitted by each modality (i.e., objects depicted in images with their descriptions in text, or spoken demon- stratives with aligned deixis via a gesture). Such disjunct mechanisms allow us to package, quantify, measure, and order our experiences, creating rich conceptual reifications and semantic differentiations. By examining the nature of these differentiations, we can study the conceptual expres- siveness of these systems (Pustejovsky, 2018). Demonstrating such knowledge is needed to ensure a shared understanding between interlocutors, and when one such interlocutor is a computer whose multichannel expres- sions are quantitatively defined, this allows us to measure certain aspects of the computational common ground cre- ated by the computer’s representation of information it has shared with its interlocutors, including humans. When two agents are co-situated and attending to the same situation (co-attending), it is the introduction of such in- formation into the discourse that creates the “shared situ- ated reference” (Pustejovsky et al., 2017) between them, and the introduction of particular information into the com- mon ground may be more or less informative depending not only on the prior contents of the common ground but also the modality through which the new …

Content truncated to 500 words