Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1823–1833, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1823 Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis Yao-Hung Hubert Tsai∗, Martin Q. Ma∗, Muqiao Yang∗, Ruslan Salakhutdinov, Louis-Philippe Morency Carnegie Mellon University, Pittsburgh, PA, USA {yaohungt, qianlim, muqiaoy, rsalakhu, lmorency}@cs.cmu.edu Abstract The human language can be expressed through multiple sources of information known as modalities, including tones of voice, facial ges- tures, and spoken language. Recent multi- modal learning with strong performances on human-centric tasks such as sentiment analy- sis and emotion recognition are often black- box, with very limited interpretability. In this paper we propose Multimodal Routing, which dynamically adjusts weights between in- put modalities and output representations dif- ferently for each input sample. Multimodal routing can identify relative importance of both individual modalities and cross-modality features. Moreover, the weight assignment by routing allows us to interpret modality- prediction relationships not only globally (i.e. general trends over the whole dataset), but also locally for each single input sample, mean- while keeping competitive performance com- pared to state-of-the-art methods. 1 Introduction The human language contains multimodal cues, in- cluding textual (e.g., spoken or written words), vi- sual (e.g., body gestures), and acoustic (e.g., voice tones) modalities. It acts as a medium for human communication and has been advanced in areas spanning affect recognition (Busso et al., 2008), media description (Lin et al., 2014), and multi- media information retrieval (Abu-El-Haija et al., 2016). Modeling multimodal sources requires to understand the relative importance of not only each single modality (defined as unimodal explanatory features) but also the interactions (defined as bi- modal or trimodal explanatory features) (B¨uchel et al., 1998). Recent work (Liu et al., 2018; Williams et al., 2018; Ortega et al., 2019) proposed methods to fuse information across modalities and ∗ indicates equal contribution. Code is avail- able at https://github.com/martinmamql/ multimodal_routing. yielded superior performance, but these models are often black-box with very limited interpretability. Figure 1: An example of Multimodal Routing, where the weights between visual, textual, and visual-textual explanatory features and concepts of emotions (happy and sad) are dynamically adjusted given every input sample. The model associates vision and v-t features to sad concept in the left sample, and v-t and text features to happy concept in the right example, showing local weights interpretation upon different input features. Interpretability matters. It allows us to identify crucial explanatory features for predictions. Such interpretability knowledge could be used to pro- vide insights into multimodal learning, improve the model design, or debug a dataset. This iner- pretability is useful at two levels: the global and the local level. The global interpretation reflects the general (averaged) trends of explanatory fea- ture importance over the whole dataset. The local interpretation is arguably harder but can give a high- resolution insight of feature importance specifically depending on each individual samples during train- ing and inference. These two levels of interpretabil- ity should provide us an understanding of unimodal, …

Content truncated to 500 words