Multi-Modal Sequence Fusion via Recursive Attention for Emotion Recognition

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 251–259 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 251 Multi-modal Sequence Fusion via Recursive Attention for Emotion Recognition Rory Beard1,∗ Ritwik Das2,∗ Raymond W. M. Ng1,∗ P. G. Keerthana Gopalakrishnan2 Luka Eerens2 Pawel Swietojanski1 Ondrej Miksik1 1Emotech Labs 2Carnegie Mellon University Abstract Natural human communication is nuanced and inherently multi-modal. Humans possess spe- cialised sensoria for processing vocal, visual, and linguistic, and para-linguistic information, but form an intricately fused percept of the multi-modal data stream to provide a holistic representation. Analysis of emotional content in face-to-face communication is a cognitive task to which humans are particularly attuned, given its sociological importance, and poses a difficult challenge for machine emulation due to the subtlety and expressive variability of cross-modal cues. Inspired by the empirical success of re- cent so-called End-To-End Memory Net- works (Sukhbaatar et al., 2015), we propose an approach based on recursive multi-attention with a shared external memory updated over multiple gated iterations of analysis. We eval- uate our model across several large multi- modal datasets and show that global contextu- alised memory with gated memory update can effectively achieve emotion recognition. 1 Introduction Multi-modal sequential data pose interesting chal- lenges for learning machines that seek to derive representations. This constitutes an increasingly relevant sub-field of multi-view learning (Ngiam et al., 2011; Baltrusaitis et al., 2017). Exam- ples of such modalities include visual, audio and textual data. Uni-modal observations are typi- cally complementary to each other and hence they can reveal a fuller and more context-rich pic- ture with better generalisation ability when used together. Through its complementary perspec- tive, each view can unburden sub-modules spe- cific to another modality of some of its modelling onus, which might otherwise learn implicit hidden ∗ Equal contribution. causes that are over-fitted to training data idiosyn- crasies in order to explain the training labels. On the other hand, multi-modal data introduces many difficulties to model designing and train- ing due to the distinct inherent dynamics of each modality. For instance, combining modalities with different temporal resolution is an open problem. Other challenges include deciding where and how modalities are combined, leveraging the weak dis- criminative power of training label and the pres- ence of variability and noise or dealing with com- plex situations such as modelling the emotion of sarcasm, where cues among modalities contradict. In this paper, we address multi-modal sequence fusion for automatic emotion recognition. We be- lieve, that a strong model should enable: (i) Specialisation of modality-specific sub- modules exploiting the inherent properties of its data stream, tapping into the mode-specific dynamics and characteristic patterns. (ii) Weak (soft) data alignment dividing heteroge- neous sequences into segments with co-occuring events across modalities without alignment to a common time axis. This overcomes limitations of hard alignments which often introduce spurious modelling assumptions and data inefficiencies (e.g. re-sampling) which must be performed again from scratch if views are added or removed. (iii) Information …

Content truncated to 500 words