Learning Paralinguistic Features from Audiobooks through Style Voice Conversion

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4736–4745 June 6–11, 2021. ©2021 Association for Computational Linguistics 4736 Learning Paralinguistic Features from Audiobooks through Style Voice Conversion Zakaria Aldeneh, Matthew Perez, Emily Mower Provost University of Michigan, Ann Arbor {aldeneh,mkperez,emilykmp}@umich.edu Abstract Paralinguistics, the non-lexical components of speech, play a crucial role in human-human interaction. Models designed to recognize paralinguistic information, particularly speech emotion and style, are difficult to train because of the limited labeled datasets available. In this work, we present a new framework that enables a neural network to learn to extract par- alinguistic attributes from speech using data that are not annotated for emotion. We as- sess the utility of the learned embeddings on the downstream tasks of emotion recognition and speaking style detection, demonstrating significant improvements over surface acous- tic features as well as over embeddings ex- tracted from other unsupervised approaches. Our work enables future systems to leverage the learned embedding extractor as a separate component capable of highlighting the paralin- guistic components of speech. 1 Introduction An effective speech-based AI system is capable of not only recognizing and interpreting the lin- guistic content of speech but also recognizing and interpreting its paralinguistic attributes. While the linguistic elements of speech encode what was said (i.e., the content), the paralinguistic el- ements encode how it was said (i.e., emotion, style, etc.) (Schuller and Batliner, 2013). The detection and modeling of paralinguistic attributes have many potential applications; ranging from affect-aware Human-Computer Interaction (HCI) systems (Vin- ciarelli et al., 2015) to the management of mental health (Karam et al., 2014; Cummins et al., 2015). One major challenge with building robust par- alinguistic models is the limited access to large- scale, labeled datasets that are needed for training the machine learning models. For instance, a typi- cal emotion dataset (e.g., IEMOCAP) that is used for building paralinguistic models contains around 12 hours of speech while a modern dataset used for building speaker recognition models contains around 2000 hours of speech (Nagrani et al., 2017). It is therefore critical that features used in paralin- guistic tasks distill relevant information from the original signal to allow the recognizers to effec- tively detect the target attributes. With this in mind, new methods that can leverage unlabeled data for distilling paralinguistic information from speech should be explored. In this work, we introduce the Expressive Voice Conversion Autoencoder (EVoCA), an unsuper- vised framework that distills paralinguistic at- tributes from speech without relying on explicit emotion or style labels. EVoCA is designed to enable a neural network to learn what it means for speech to be expressive by treating expressive speech as a modulation of neutral speech. EVoCA is trained using parallel speech inputs: one expres- sive and one neutral. However, these types of paral- lel paralinguistic corpora are not available at scale. To address this, we use a large audiobook corpus (i.e., 200 hours) composed of expressive speech and artificially generate the parallel …

Content truncated to 500 words