Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4619–4629 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4619 Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper) Santiago Castro†∗, Devamanyu HazarikaΦ∗, Verónica Pérez-Rosas†, Roger ZimmermannΦ, Rada Mihalcea†, Soujanya Poriaι †Computer Science & Engineering, University of Michigan, USA ΦSchool of Computing, National University of Singapore, Singapore ιInformation Systems Technology and Design, SUTD, Singapore {sacastro,vrncapr,mihalcea}@umich.edu, {hazarika,rogerz}@comp.nus.edu.sg, sporia@sutd.edu.sg Abstract Sarcasm is often expressed through several ver- bal and non-verbal cues, e.g., a change of tone, overemphasis in a word, a drawn-out syllable, or a straight looking face. Most of the re- cent work in sarcasm detection has been car- ried out on textual data. In this paper, we ar- gue that incorporating multimodal cues can im- prove the automatic classification of sarcasm. As a first step towards enabling the develop- ment of multimodal approaches for sarcasm detection, we propose a new sarcasm dataset, Multimodal Sarcasm Detection Dataset (MUS- tARD1), compiled from popular TV shows. MUStARD consists of audiovisual utterances annotated with sarcasm labels. Each utterance is accompanied by its context of historical ut- terances in the dialogue, which provides addi- tional information on the scenario where the ut- terance occurs. Our initial results show that the use of multimodal information can reduce the relative error rate of sarcasm detection by up to 12.9% in F-score when compared to the use of individual modalities. The full dataset is pub- licly available for use at https://github. com/soujanyaporia/MUStARD. 1 Introduction Sarcasm plays an important role in daily conversa- tions by allowing individuals to express their intent to mock or display contempt. It is achieved by us- ing irony that reflects a negative connotation. For example, in the utterance: Maybe it’s a good thing we came here. It’s like a lesson in what not to do, the sarcasm is explicit as the speaker expresses learning of a lesson in a positive light when in real- ity, she means it in a negative way. However, there are also scenarios where sarcasm lacks explicit lin- guistic markers, thus requiring additional cues that ∗Equal contribution. 1MUStARD is an abbreviation for MUltimodal SARcasm Dataset. Similar to how “mustard” adds spice to our food and meals, we believe sarcasm adds spice to our interactions and lives. can reveal the speaker’s intentions. For instance, sarcasm can be expressed using a combination of verbal and non-verbal cues, such as a change of tone, overemphasis in a word, a drawn-out sylla- ble, or a straight looking face. Moreover, sarcasm detection involves finding linguistic or contextual incongruity, which in turn requires further informa- tion, either from multiple modalities (Schifanella et al., 2016; Mishra et al., 2016a) or from the con- text history in a dialogue. This paper explores the role of multimodality and conversational context in sarcasm detection and introduces a new resource to further enable research in this area. More specifically, our paper makes the following contributions: (1) We curate a new dataset, MUStARD, for multimodal …

Content truncated to 500 words