Towards end-2-end learning for predicting behavior codes from spoken utterances in psychotherapy conversations

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3797–3803 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3797 Towards End-2-end Learning for Predicting Behavior Codes from Spoken Utterances in Psychotherapy Conversations Karan Singla1, Zhuohao Chen1, David C. Atkins2, Shrikanth Narayanan1 1University of Southern California, Los Angeles, USA 2University of Washington, Seattle, WA, USA {singlak, zhuohaoc}@usc.edu, datkins@uw.edu, shri@sipi.usc.edu Abstract Spoken language understanding tasks usu- ally rely on pipelines involving complex pro- cessing blocks such as voice activity detec- tion, speaker diarization and Automatic speech recognition (ASR). We propose a novel frame- work for predicting utterance level labels di- rectly from speech features, thus removing the dependency on first generating transcripts, and transcription free behavioral coding. Our clas- sifier uses a pretrained Speech-2-Vector en- coder as bottleneck to generate word-level rep- resentations from speech features. This pre- trained encoder learns to encode speech fea- tures for a word using an objective similar to Word2Vec. Our proposed approach just uses speech features and word segmentation infor- mation for predicting spoken utterance-level target labels. We show that our model achieves competitive results to other state-of-the-art ap- proaches which use transcribed text for the task of predicting psychotherapy-relevant be- havior codes. 1 Introduction Speech interfaces have seen a widely growing trend and this has brought about increasing interest in advancing computational approaches to spoken lan- guage understanding (SLU). (Tur and De Mori, 2011; Xu and Sarikaya, 2014; Yao et al., 2013; Ravuri and Stolcke, 2015). SLU systems often rely on Automatic speech recognition (ASR) for gen- erating lexical features. The ASR output is then used for the target natural language understanding task. Furthermore, end-2-end SLU systems for var- ious applications, including speech synthesis (Oord et al., 2016), ASR tasks (Amodei et al., 2016; Chan et al., 2016; Soltau et al., 2016) and speech-2-text translation (Chung et al., 2019) have shown promis- ing results. Recently (Haque et al., 2019) propose a method for learning audio-linguistuc embedding but that too depends on using transcribed text. Pretrained word embeddings ASR Transcript Word- segmentation Speech-2- Vector encoder Speech features Previous works Our approach Behavior code predictor Figure 1: Upper part describes most of existing ap- proaches which either use ASR or manual transcripts. Lower part shows our proposed approach where we pre- dict behavior codes without using transcripts Due to the nature of the speech processing pipeline, natural language understanding tasks suf- fer from two major problems, 1) error propagation through ASR leading to noisy lexical features 2) loss of rich information which supplement lexical features, such as prosodic and acoustic expressive speech patterns. In this paper, we propose a framework to ad- dress the problem of predicting behavior codes di- rectly from speech utterances. We focus on data from Motivational Interviewing (MI) sessions, a type of talk-based psychotherapy focused on be- havior change. In psychology research and clinical practice, behavioral coding is often used to under- stand process mechanisms and therapy efficacy and outcomes. Behavior codes are annotated by an expert …

Content truncated to 500 words