JTAV: Jointly Learning Social Media Content Representation by Fusing Textual, Acoustic, and Visual Features

Proceedings of the 27th International Conference on Computational Linguistics, pages 1269–1280 Santa Fe, New Mexico, USA, August 20-26, 2018. 1269 JTAV: Jointly Learning Social Media Content Representation by Fusing Textual, Acoustic, and Visual Features Hongru Liang1, Haozheng Wang1, Jun Wang2, Shaodi You3, Zhe Sun4, Jin-Mao Wei1, Zhenglu Yang1∗ 1 CCCE, Nankai University, Tianjin, China 2 College of Mathematics and Statistics Science, Ludong University, China 3 Data61-CSIRO, Australian National University, Canberra, Australia 4 RIKEN Head Office for Information Systems and Cybersecurity Computational Engineering Applications Unit, Japan {lianghr, hzwang, junwang}@mail.nankai.edu.cn, shaodi.you@data61.csiro.au, zhe.sun.vk@riken.jp, {weijm, yangzl}@nankai.edu.cn Abstract Learning social media content is the basis of many real-world applications, including information retrieval and recommendation systems, among others. In contrast with previous works that focus mainly on single modal or bi-modal learning, we propose to learn social media content by fus- ing jointly textual, acoustic, and visual information (JTAV). Effective strategies are proposed to extract fine-grained features of each modality, that is, attBiGRU and DCRNN. We also introduce cross-modal fusion and attentive pooling techniques to integrate multi-modal information com- prehensively. Extensive experimental evaluation conducted on real-world datasets demonstrates our proposed model outperforms the state-of-the-art approaches by a large margin. 1 Introduction The popularity of the social media (e.g., Twitter, Weibo) over the last two decades has led to increas- ing demands for learning the content of social media, which may be beneficial to many real-world ap- plications, such as sentiment analysis (Wang et al., 2015), information retrieval (Li et al., 2017), and recommendation systems (Wu et al., 2017). However, the task is non-trivial and challenging, because social media content is commonly multi-modal and involves several types of data, including text, audio, and image (an example is illustrated in Fig. 1). Each representation of the individual modality encodes specific knowledge and complementary, which can be explored to facilitate understanding of the entire meaning of the content. Therefore, gaining comprehensive knowledge from learning arising out of so- cial media requires deliberate exploration of single-modal information and joint learning of the intrinsic correlation among various-modalities. Figure 1: An example of multi-modal social media content, organized by a textual passage, a clip of the song “Muhammad Ali”, and a picture of Muhammad Ali’s famous victory. This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0/ ∗Corresponding author 1270 Previous works on learning social media content have focused mainly on single modality, e.g., textual information (Wu et al., 2017) and image information (Garimella et al., 2016). To our best knowledge, no studies have focused on mere acoustic information analysis for social media. These previous works cope with data in the early stage of social media, because users are limited to posting single-modal content. In contrast, today’s social media content usually includes textual, acoustic, and visual ingredients simul- taneously, and requires multi-modal learning strategies to integrate rich information from all modalities. Many researchers have studied the bi-modal utilization to address the issue of incomplete information exploration. This utilization can be classified into two kinds of approaches. The …

Content truncated to 500 words