Towards Sentiment and Emotion aided Multi-modal Speech Act Classification in Twitter

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5727–5737 June 6–11, 2021. ©2021 Association for Computational Linguistics 5727 Towards Sentiment and Emotion aided Multi-modal Speech Act Classification in Twitter Tulika Saha, Apoorva Upadhyaya, Sriparna Saha and Pushpak Bhattacharyya Department of Computer Science and Engineering, Indian Institute of Technology Patna, India (sahatulika15,sriparna.saha,pushpakbh)@gmail.com Abstract Speech Act Classification determining the communicative intent of an utterance has been investigated widely over the years as a stan- dalone task. This holds true for discussion in any fora including social media platform such as Twitter. But the emotional state of the tweeter which has a considerable effect on the communication has not received the at- tention it deserves. Closely related to emo- tion is sentiment, and understanding of one helps understand the other. In this work, we firstly create a new multi-modal, emotion- TA (’TA’ means tweet act, i.e., speech act in Twitter) dataset called EmoTA collected from open-source Twitter dataset. We propose a Dyadic Attention Mechanism (DAM) based multi-modal, adversarial multi-tasking frame- work. DAM incorporates intra-modal and inter-modal attention to fuse multiple modal- ities and learns generalized features across all the tasks. Experimental results indicate that the proposed framework boosts the per- formance of the primary task, i.e., TA clas- sification (TAC) by benefitting from the two secondary tasks, i.e., Sentiment and Emotion Analysis compared to its uni-modal and single task TAC (tweet act classification) variants. 1 Introduction Identification of speech acts is one of the prelimi- nary means of determining the communicative in- tent or pragmatics of a speaker (for example, state- ment, request, question etc.). This is true for dia- logue system, speech transcription, social media such as Twitter, MySpace etc. Twitter is one of the leading micro-blogging services. By 2019, 330 million users were active monthly and 500 mil- lion tweets were sent per day1. Identification of tweet acts (TAs- speech acts in Twitter) is highly beneficial for Twitter as well as tweeters. For Twit- ter, it helps decipher a particular subject in terms 1https://www.omnicoreagency.com/twitter-statistics/ of speech acts and discrepancy identification. It also helps in social media monitoring by analysing topic alteration or spamming. It assists the follow- ers in monitoring and scanning the subject with the most advantageous speech acts based on their needs. This helps reduce their search space and encourages them to obtain useful information from out of millions of tweets. It gives the tweeter a greater sense of the content, mood and trend. A person’s emotional state and sentiment greatly impacts its intended content (Barrett et al., 1993). Often sentiment and emotion are treated as two dif- ferent problems (Do et al., 2019), (Soleymani et al., 2017), (Albanie et al., 2018), (Hossain and Muham- mad, 2019), (Majumder et al., 2019). However, sentiment and emotion are are closely related. For example, emotions such as happy and joy are inher- ently related to a positive sentiment. But emotion is much more nuanced and fine-grained compared to sentiment (Kumar …

Content truncated to 500 words