MONAH: Multi-Modal Narratives for Humans to analyze conversations

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 466–479 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 466 MONAH: Multi-Modal Narratives for Humans to analyze conversations Joshua Y. Kim Greyson Y. Kim Chunfeng Liu University of Sydney Success Beyond Pain Hello Sunday Morning New South Wales, Australia Western Australia, Australia New South Wales, Australia josh.kim@sydney.edu.au greyson.kim@gmail.com ian@hellosundaymorning.org Rafael A. Calvo Silas C.R. Taylor Kalina Yacef∗ Imperial College London University of New South Wales University of Sydney London, United Kingdom New South Wales, Australia New South Wales, Australia r.calvo@imperial.ac.uk silas.taylor@unsw.edu.au kalina.yacef@sydney.edu.au Abstract In conversational analyses, humans manually weave multimodal information into the tran- scripts, which is significantly time-consuming. We introduce a system that automatically expands the verbatim transcripts of video- recorded conversations using multimodal data streams. This system uses a set of prepro- cessing rules to weave multimodal annotations into the verbatim transcripts and promote in- terpretability. Our feature engineering contri- butions are two-fold: firstly, we identify the range of multimodal features relevant to de- tect rapport-building; secondly, we expand the range of multimodal annotations and show that the expansion leads to statistically significant improvements in detecting rapport-building. 1 Introduction Dyadic human-human dialogs are rich in multi- modal information. Both the visual and the audio characteristics of how the words are said reveal the emotions and attitudes of the speaker. Given the richness of multimodal information, analyzing conversations requires both domain knowledge and time. The discipline of conversational analysis is a mature field. In this discipline, conversations could be manually transcribed using a technical system developed by Jefferson (2004), containing informa- tion about intonation, lengths of pauses, and gaps. Hence, it captures both what was said and how it was said1. However, such manual annotations take a great deal of time. Individuals must watch the conversations attentively, often replaying the conversations to ensure completeness. Automated Jefferson (2004) transcripts could be generated from video-recordings (Moore, 2015). * Corresponding author 1Please visit www.universitytranscriptions.co.uk/jefferson- transcription-example/ for an audio example. However, the potential issue with Jeffersonian an- notations is that there are often within-word anno- tations and symbols which makes it hard to benefit from pre-trained word embeddings. Inspired by the Jeffersonian annotations, we expand the verba- tim transcripts with multimodal annotations such that downstream classification models can easily benefit from pre-trained word embeddings. Our paper focuses on the classification task of predicting rapport building in conversations. Rap- port has been defined as a state experienced in in- teraction with another with interest, positivity, and balance (Cappella, 1990). If we can model rapport building in the medical school setting, the volun- teer actors can let the system give feedback for the unofficial practice sessions, and therefore students get more practice with feedback. Also, the lecturer could study the conversations of the top perform- ers and choose interesting segments to discuss. As student doctors get better in rapport building, when they graduate and practice as doctors, treatments are more effective and long-term (Egbert et al., 1964; …

Content truncated to 500 words