Strong and Simple Baselines for Multimodal Utterance Embeddings

Proceedings of NAACL-HLT 2019, pages 2599–2609 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2599 Strong and Simple Baselines for Multimodal Utterance Embeddings Paul Pu Liang∗, Yao Chong Lim∗, Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, Louis-Philippe Morency School of Computer Science, Carnegie Mellon University {pliang,yaochonl,yaohungt,rsalakhu,morency}@cs.cmu.edu Abstract Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intonations. Learning representations for these spoken utterances is a complex research problem due to the presence of multiple heterogeneous sources of informa- tion. Recent advances in multimodal learning have followed the general trend of building more complex models that utilize various at- tention, memory and recurrent components. In this paper, we propose two simple but strong baselines to learn embeddings of multimodal utterances. The first baseline assumes a con- ditional factorization of the utterance into uni- modal factors. Each unimodal factor is mod- eled using the simple form of a likelihood function obtained via a linear transformation of the embedding. We show that the opti- mal embedding can be derived in closed form by taking a weighted average of the unimodal features. In order to capture richer represen- tations, our second baseline extends the first by factorizing into unimodal, bimodal, and tri- modal factors, while retaining simplicity and efficiency during learning and inference. From a set of experiments across two tasks, we show strong performance on both supervised and semi-supervised multimodal prediction, as well as significant (10 times) speedups over neural models during inference. Overall, we believe that our strong baseline models offer new benchmarking options for future research in multimodal learning. 1 Introduction Human language is a rich multimodal signal con- sisting of spoken words, facial expressions, body gestures, and vocal intonations (Streeck and Knapp, 1992). At the heart of many multimodal modeling tasks lies the challenge of learning rich represen- tations of spoken utterances from multiple modal- ities (Papo et al., 2014). However, learning repre- * authors contributed equally sentations for these spoken utterances is a complex research problem due to the presence of multiple heterogeneous sources of information (Baltruˇsaitis et al., 2017). This challenging yet crucial research area has real-world applications in robotics (Mon- talvo et al., 2017; Noda et al., 2014), dialogue sys- tems (Johnston et al., 2002; Rudnicky, 2005), intel- ligent tutoring systems (Mao and Li, 2012; Banda and Robinson, 2011; Pham and Wang, 2018), and healthcare diagnosis (Wentzel and van der Geest, 2016; Lisetti et al., 2003; Sonntag, 2017). Recent progress on multimodal representation learning has investigated various neural models that utilize one or more of attention, memory and recurrent compo- nents (Yang et al., 2017; Liang et al., 2018). There has also been a general trend of building more com- plicated models for improved performance. In this paper, we propose two simple but strong baselines to learn embeddings of multimodal utter- ances. The first baseline assumes a factorization of the utterance into unimodal factors conditioned on the joint embedding. Each unimodal factor is mod- eled using …

Content truncated to 500 words