Modulating Language Models with Emotions

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4332–4339 August 1–6, 2021. ©2021 Association for Computational Linguistics 4332 Modulating Language Models with Emotions Ruibo Liu1 Jason Wei2 Chenyan Jia3 Soroush Vosoughi1 1Dartmouth College 2ProtagoLabs 3University of Texas at Austin ruibo.liu.gr@dartmouth.edu jason@protagolabs.com chenyanjia@utexas.edu soroush@dartmouth.edu Abstract Generating context-aware language that em- bodies diverse emotions is an important step towards building empathetic NLP systems. In this paper, we propose a formulation of mod- ulated layer normalization—a technique in- spired by computer vision—that allows us to use large-scale language models for emo- tional response generation. In automatic and human evaluation on the MojiTalk dataset, our proposed modulated layer normalization method outperforms prior baseline methods while maintaining diversity, fluency, and co- herence. Our method also obtains competitive performance even when using only 10% of the available training data. 1 Introduction Building interactive systems that can understand and express human emotions has been a long-term goal of artificial intelligence (Shen and Feng, 2020; Huang et al., 2018; Salovey and Sluyter, 1997). Given a context, an intelligent agent ought to be able to generate responses that not only consider the context but also reflect a specified emotion, a task called emotional response generation. One common representation of emotions is through emojis, which often convey the underlying emo- tions in an utterance (Zhou and Wang, 2018). Table 1 shows an example generation in this formulation. To tackle this problem, prior work has proposed a number of different models, including variants of sequence-to-sequence (Seq2Seq) models (Serban et al., 2016; Li et al., 2016a), variational autoen- coders (VAE) (Gu et al., 2019; Shen et al., 2017; Zhao et al., 2017) and adversarial networks (Kong et al., 2019; Li et al., 2017). Their generated re- sponses are often dull or generic, partially due to the limited training data for diverse emotions (Li et al., 2017). More recent studies have tried to Context: Emotion Response good game start morning off tigers v eagles. good luck to all the eagles i m not a tigers fan but we ve got a win we ve got to wait for tommorrow for the game hope you enjoyed the match with your team Table 1. Example generation of our method for four different emojis. Context is an actual random tweet, and emotion is specified by emojis. pre-train language models (LMs) on specific do- main data to pivot generation towards certain di- rection (Gao et al., 2020; Zhang et al., 2020; Yang et al., 2020; Keskar et al., 2019). However, training a LM from scratch can be costly, and collecting sufficient pre-training data in diverse emotions is also challenging, especially for low-resource emo- tions (Yang et al., 2019a). In this work, we present a simple and easy- to-deploy technique that can enable pre-trained large-scale LMs to generate fine-grained emotional responses. Specifically, we inject emotional sig- nals specified by 64 commonly used emojis via Modulated Layer Normalization (Mod-LN), a tech- nique widely adopted in computer vision but whose potential has not been well studied yet in …

Content truncated to 500 words