Teaching Neural Module Networks to Do Arithmetic

Proceedings of the 29th International Conference on Computational Linguistics, pages 134–140 October 12–17, 2022. 134 Gestures Are Used Rationally: Information Theoretic Evidence from Neural Sequential Models Yang Xu Department of Computer Science San Diego State University yang.xu@sdsu.edu Yang Cheng University of Southern California ycheng04@usc.edu Riya Bhatia Carnegie Mellon University riyabhat@andrew.cmu.edu Abstract Verbal communication is companied by rich non-verbal signals. The usage of gestures, poses, and facial expressions facilitates the information transmission in verbal channel. However, few computational studies have ex- plored the non-verbal channels with finer theo- retical lens. We extract gesture representations from monologue video data and train neural se- quential models, in order to study the degree to which non-verbal signals can effectively transmit information. We focus on examining whether the gestures demonstrate the similar pattern of entropy rate constancy (ERC) found in words, as predicted by Information Theory. Positive results are shown to support the as- sumption, which leads to the conclusion that speakers indeed use simple gestures to convey information that enhances verbal communica- tion, and the production of non-verbal informa- tion is rationally organized. 1 Introduction Communication is a multi-modal process, in which information from verbal and non-verbal modalities are mixed into one channel. It has been revealed from a long history of empirical studies that speak- ers’ expression in visual modality, including ges- tures, body poses, eye contacts and other types of non-verbal behaviors, play critical roles in face-to- face communication, as they add subtle information that is hard to convey in verbal language (Pease and Pease, 2008; Krauss et al., 1996). However, it re- mains an untested idea to view these sparse and random non-verbal signals as a formal communica- tion channel that transmits “serious” information, which has seldom been validated by computational studies. A key missing step is to explore whether the non-verbal information can be quantified. The questions that are worth further investigation include: How rich is the information contained in these non-verbal channels? What are their relation- ships to verbal information? Can we understand the meanings of different gestures, poses, and motions embedded in spontaneous language in a similar way to understanding word meanings? The goal of this study is to take a simple yet necessary first step approaching the above questions, by examining a basic Information Theoretic property of gestures that comes along with verbal language. Some pre- liminary but prospective results are presented. 2 Related Work 2.1 Gestures as non-verbal communication There is vast literature on the connection between gesture and language in human communication. Gestures, defined as “the spontaneous hand move- ments produced in rhythm with speech” (Clough and Duff, 2020) naturally co-occur with spoken language. According to the thorough survey from (Clough and Duff, 2020), the communication func- tion of gestures is one of the main focus of early studies. McNeill (1992) has classified ges- tures into two categories, representative and non- representative, in which the former has clearer se- mantic meanings (e.g., depicting objects and de- scribing locations), while the latter refers to the brief, …

Content truncated to 500 words