A New View of Multi-modal Language Analysis: Audio and Video Features as Text “Styles”

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1956–1965 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1956 A New View of Multi-modal Language Analysis: Audio and Video Features as Text “Styles” Zhongkai Sun University of Wisconsin-Madison zsun227@wisc.edu Prathusha K Sarma∗ Apple prathyushaks.21@gmail.com Yingyu Liang University of Wisconsin-Madison yliang@cs.wisc.edu William A. Sethares University of Wisconsin-Madison sethares@wisc.edu Abstract Imposing the style of one image onto another is called style transfer. For example, the style of a Van Gogh painting might be imposed on a photograph to yield an interesting hybrid. This paper applies the adaptive normalization used for image style transfer to language semantics, i.e., the style is the way the words are said (tone of voice and facial expressions) and these are style-transferred onto the text. The goal is to learn richer representations for multi-modal utterances using style-transferred multi-modal features. The proposed Style-Transfer Trans- former (STT) grafts a stepped styled adaptive layer-normalization onto a transformer net- work, the output from which is used in senti- ment analysis and emotion recognition prob- lems. In addition to achieving performance on par with the state-of-the art (but using less than a third of the model parameters), we examine the relative contributions of each mode when used in the downstream applications. 1 Introduction Multi-modal language analysis expands textual analysis by utilizing co-occurring acoustic and vi- sual information, and has recently become a popu- lar topic in machine learning (Morency et al., 2011; Baltruˇsaitis et al., 2018). In both sentiment analysis (Wang et al., 2016; Zadeh et al., 2016) and emo- tion recognition (Busso et al., 2008; Mittal et al., 2019), the three modalities are combined to better represent the sentiment or emotional meaning of a passage. The idea of combing textual, acoustic, and visual features is obvious: individual modalities are not always able to convey as accurate an impression as multi-modal features, which typically provide more complete information. For instance, Fig. 1 ∗Work done while at UW-Madison shows a caricature where the sentiment may be eas- ily understood from the audio and video, but not from textual analysis alone. Figure 1: An example of multi-modal sentiment anal- ysis. In this example, textual information is unrelated to the underlying sentiment while acoustic and visual features reflect the perceived sentiment. Combining textual, acoustic, and visual features can be accomplished in a variety of ways ((Tsai et al., 2018; Zadeh et al., 2018a; Liang et al., 2018; Mai et al., 2019; Sun et al., 2019b; Mittal et al., 2019)). Among these, (Wang et al., 2018b; Sun et al., 2019a) consider text as the backbone and study methods to inject acoustic and visual infor- mation into the textual features that are typically extracted via pre-trained word/language models, e.g., Glove (Pennington et al., 2014), ELMO (Pe- ters et al., 2018), and BERT (Devlin et al., 2018). This paper also uses textual features as a back- bone and studies a novel way of injecting non-text features into a primarily text-only …

Content truncated to 500 words