FRUIT: Faithfully Reflecting Updated Information in Text

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 336 - 347 July 10-15, 2022 ©2022 Association for Computational Linguistics Empathic Machines: Using Intermediate Features as Levers to Emulate Emotions in Text-To-Speech Systems Saiteja Kosgi 1 Sarath Sivaprasad1,2 Niranjan Pedanekar 2 Anil Nelakanti3 Vineet Gandhi1 1Kohli Centre on Intelligent Systems, IIIT Hyderabad 2TCS Research, Pune 3Prime Video, Amazon Bengaluru {saiteja.k,sarath.s}@research.iiit.ac.in,n.pedanekar@tcs.com annelaka@amazon.com, vgandhi@iiit.ac.in Abstract We present a method to control the emotional prosody of Text to Speech (TTS) systems by us- ing phoneme-level intermediate features (pitch, energy, and duration) as levers. As a key idea, we propose Differential Scaling (DS) to dis- entangle features relating to affective prosody from those arising due to acoustics conditions and speaker identity. With thorough exper- imental studies, we show that the proposed method improves over the prior art in accu- rately emulating the desired emotions while retaining the naturalness of speech. We ex- tend the traditional evaluation of using indi- vidual sentences for a more complete evalua- tion of HCI systems. We present a novel ex- perimental setup by replacing an actor with a TTS system in offline and live conversa- tions. The emotion to be rendered is either predicted or manually assigned. The results show that the proposed method is strongly pre- ferred over the state-of-the-art TTS system and adds the much-coveted “human touch” in ma- chine dialogue. Audio samples for our exper- iments and the code are available at: https: //emtts.github.io/tts-demo/ 1 Introduction “The text is like a canoe, and the river on which it sits is the emotion. It all depends on the flow of the river, which is your emotion. The text takes on the character of your emotion.” — Sanford Meisner In natural language processing, vocabulary and grammar tend to take center stage, but those ele- ments of speech only tell half the story. Affective prosody provides context and gives meaning to words, and keeps listeners engaged. Understand- ing emotional prosody is central to language and social development. Studies suggest that we show remarkable sensitivity to prosody "even as infants" (Nazzi et al., 1998; Massicotte-Laforge and Shi, 2015). Recently Kraus (2017) shows that voice- only communication likely elicits higher empathic Do you love me Juliet Juliet Romeo What do you think I know what you are thinking Do you love me What do you think I know what you are thinking Juliet Juliet Romeo Figure 1: Dialogues can have different meanings de- spite having the same text. Also, starting with the same emotion, Juliet has different emotions post Romeo’s response. accuracy than even multi-sense modes including facial expressions. Buchholz (2016) shows that any meaningful spo- ken dialogue cannot happen without some amount of prosodic matching. As humans, we naturally an- ticipate and adapt with emotional cues in convers- ing with others, see Figure 1 for an example. Cele- brated trainer Sanford Meisner employed this to de- velop Meisner technique for theatre actors to react naturally to others in the …

Content truncated to 500 words