Deep Learning of Audio and Language Features for Humor Prediction

Deep Learning of Audio and Language Features for Humor Prediction Dario Bertero, Pascale Fung Human Language Technology Center Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong dbertero@connect.ust.hk, pascale@ece.ust.hk Abstract We propose a comparison between various supervised machine learning methods to predict and detect humor in dialogues. We retrieve our humorous dialogues from a very popular TV sitcom: “The Big Bang Theory”. We build a corpus where punchlines are annotated using the canned laughter embedded in the audio track. Our comparative study involves a linear-chain Conditional Random Field over a Recurrent Neural Network and a Convolutional Neural Network. Using a combination of word-level and audio frame-level features, the CNN outperforms the other methods, obtaining the best F-score of 68.5% over 66.5% by CRF and 52.9% by RNN. Our work is a starting point to developing more effective machine learning and neural network models on the humor prediction task, as well as developing machines capable in understanding humor in general. Keywords: humor prediction, neural networks, TV-sitcoms 1. Introduction The term “humor” refers to various kinds of stimuli, includ- ing acoustic, verbal, visual and situational, that are able to trigger a laughter reaction in the recipient. It is an important aspect of our everyday life, and is supposed to give benefits to physical and psychological health (Sumners, 1988; Mar- tineau, 1972; La Fave et al., 1976; Anderson and Arnoult, 1989; Lefcourt et al., 1997; Lefcourt and Martin, 2012). There has recently been many attempts in detecting humor from canned jokes (Yang et al., 2015), customer reviews (Reyes and Rosso, 2012) and Twitter (Reyes et al., 2013; Barbieri and Saggion, 2014; Riloff et al., 2013; Joshi et al., 2015). All these analyses are only on isolated textual data. Fewer work took into consideration other elements, such as the surrounding context (Bamman and Smith, 2015; Karoui et al., 2015) or acoustic and prosodic features (Rakov and Rosenberg, 2013). We propose to predict when people would laugh in a dialog with a supervised machine learning approach. While most of the past attempts concentrate on isolated examples, the response to humor in a conversation depends heavily on the surrounding context, such as the conversational topic and the previous utterances. It is quite common that the same utterance may trigger a different effect on the recipient de- pending on when it is used. Two distinct moments can be identified in humor and joke generation: a setup where ap- propriate inputs are given and the context for the joke is built, and the “punchline” where the climax is reached and people are triggered to react with laugh (Hetzron, 1991; At- tardo, 1997). Our task is to identify these punchlines and thus predict where laughter occurs in the dialog flow. More- over the way a spoken dialog utterance is made is another important element that may trigger a humorous reaction. Thus we also propose to combine acoustic and language features. To meet our objectives we build a corpus with …

Content truncated to 500 words