Textless Speech Emotion Conversion using Discrete & Decomposed Representations

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11200 - 11214 December 7-11, 2022 ©2022 Association for Computational Linguistics Textless Speech Emotion Conversion using Discrete & Decomposed Representations Felix Kreuk1,2∗, Adam Polyak2, Jade Copet2, Eugene Kharitonov2, Tu-Anh Nguyen2, Morgane Rivière2, Wei-Ning Hsu2, Abdelrahman Mohamed2, Emmanuel Dupoux2,3, Yossi Adi2 Bar-Ilan University, Ramat-Gan, Israel1 Meta AI Research2 EHESS, Paris3 {felixkreuk, adiyoss}@fb.com Abstract Speech emotion conversion is the task of mod- ifying the perceived emotion of a speech utterance while preserving the lexical con- tent and speaker identity. In this study, we cast the problem of emotion conversion as a spoken language translation task. We use a decomposition of the speech signal into discrete learned representations, consist- ing of phonetic-content units, prosodic fea- tures, speaker, and emotion. First, we modify the speech content by translating the phonetic- content units to a target emotion, and then predict the prosodic features based on these units. Finally, the speech waveform is gener- ated by feeding the predicted representations into a neural vocoder. Such a paradigm al- lows us to go beyond spectral and parametric changes of the signal, and model non-verbal vocalizations, such as laughter insertion, yawn- ing removal, etc. We demonstrate objectively and subjectively that the proposed method is vastly superior to current approaches and even beats text-based systems in terms of perceived emotion and audio quality. We rigorously eval- uate all components of such a complex system and conclude with an extensive model analysis and ablation study to better emphasize the ar- chitectural choices, strengths and weaknesses of the proposed method. Samples are available under the following link: [samples]. 1 Introduction Generating spoken utterances and dialogue that sound natural is a fundamental requirement to improve human-computer interaction (Tits et al., 2019). One of the main roadblock in improving nat- uralness in speech generation is the modeling of ex- pressive and emotional states. The difficulty is that emotion is a phenomenon affecting all linguistic levels simultaneously: when one goes from a happy ∗ Work done while Felix Kreuk was an Intern at Meta AI Research. Figure 1: An illustration of the proposed system. The input signal is first encoded as a discrete sequence of content units (Ec). Next, a sequence to sequence (S2S) model is applied to translate between the sequences cor- responding to different emotions. Then we predict the duration (Edur) and F0 (EF 0) before feeding these sig- nals to a vocoder (G). The last 4 components are condi- tioned by emotion (zemo). We use pink to denote mod- els and green to denote representations. to an angry state, one may use different vocabu- lary, insert non-verbal vocalizations (cries, grunts, etc), modify prosody (intonation and rhythm), and change voice quality due to stress. Vice versa each of the levels contributes to the perception of the emotional state of the speaker, where the non verbal aspects can often override the lexical content (Mehrabian and Wiener, 1967). Existing emotion generation or emotion con- version techniques have a hard time producing …

Content truncated to 500 words