Box of Lies: Multimodal Deception Detection in Dialogues

Proceedings of NAACL-HLT 2019, pages 1768–1777 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1768 Box of Lies R⃝: Multimodal Deception Detection in Dialogues Felix Soldner Department of Security and Crime Science University College London felix.soldner@ucl.ac.uk Ver´onica P´erez-Rosas and Rada Mihalcea Department of Computer Science and Engineering University of Michigan, Ann Arbor {vrncapr,mihalcea}@umich.edu Abstract Deception often takes place during everyday conversations, yet conversational dialogues re- main largely unexplored by current work on automatic deception detection. In this pa- per, we address the task of detecting mul- timodal deceptive cues during conversational dialogues. We introduce a multimodal dataset containing deceptive conversations between participants playing The Tonight Show Star- ring Jimmy Fallon R ⃝ Box of Lies game, in which they try to guess whether an object de- scription provided by their opponent is decep- tive or not. We conduct annotations of mul- timodal communication behaviors, including facial and linguistic behaviors, and derive sev- eral learning features based on these annota- tions. Initial classification experiments show promising results, performing well above both a random and a human baseline, and reaching up to 69% accuracy in distinguishing decep- tive and truthful behaviors. 1 Introduction Deception occurs often during dialogues, but un- til now this setting has received little attention from the research community (Tsunomori et al., 2015). In this paper, we explore verbal, non- verbal, and conversational dialog cues between contestants playing the Box of Lies game in The Tonight Show Starring Jimmy Fallon R⃝ tv show. In the game, participants try to guess whether an ob- ject description provided by their opponent is de- ceptive or not. The game scenario provides a rich environment where we can explore several aspects of deceptive behavior occurring during conversa- tions. First, it allows us to study conversational deception in the presence of multiple modalities such as verbal and non-verbal behaviors. Second, it provides observable assessments of participant’s honesty, which is usually an important challenge during deception research. Third, since partici- pants experience the pressure to win the game in front of a big audience, it presumably presents an environment with high stakes.1 Recent work on multimodal deception detection has already shown the importance of verbal and non-verbal behaviors during the automatic iden- tification of deceit (Abouelenien et al., 2017a). Following this line of work, our main contribu- tion consists of investigating whether such modal- ities can also be leveraged to predict deception in a conversational dialog as well as exploring whether the dialogue setting adds meaningful in- formation to the other modalities to potentially in- crease the classification performance. Based on earlier work (Perez-Rosas et al., 2014; Mihalcea et al., 2013), we hypothesize that (1) including dialogue features in addition to other multimodal features (language and facial expressions) while training a classifier increases the prediction perfor- mance; (2) automatic classification of truthful and deceptive behavior is better than random guess- ing (50% for equal class sizes); and (3) automatic classification of truthful and deceptive responses is more accurate than human …

Content truncated to 500 words