Learning Word Ratings for Empathy and Distress from Document-Level User Responses

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1664–1673 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1664 Learning Word Ratings for Empathy and Distress from Document-Level User Responses Jo˜ao Sedoc*♣, Sven Buechel*†♠, Yehonathan Nachmany♦, Anneke Buffone♦, and Lyle Ungar♦ ♣Johns Hopkins University, ♠Friedrich-Schiller-Universit¨at Jena, ♦University of Pennsylvania jsedoc@jhu.edu, sven.buechel@uni-jena.de, yoninachmany@gmail.com, buffonea@sas.upenn.edu, ungar@cis.upenn.edu Abstract Despite the excellent performance of black box approaches to modeling sentiment and emotion, lexica (sets of informative words and associated weights) that characterize different emotions are indispensable to the NLP community because they allow for interpretable and robust predictions. Emotion analysis of text is increasing in popularity in NLP; however, manually creating lexica for psychological constructs such as empathy has proven difficult. This paper automatically creates empathy word ratings from document-level ratings. The underlying problem of learning word ratings from higher-level supervision has to date only been addressed in an ad hoc fashion and has not used deep learning methods. We systematically compare a number of approaches to learning word ratings from higher-level supervision against a Mixed-Level Feed Forward Network (MLFFN), which we find performs best, and use the MLFFN to create the first-ever empathy lexicon. We then use Signed Spectral Clustering to gain insights into the resulting words. The empathy and distress lexica are publicly available at: http://www.wwbp.org/lexica.html. Keywords: lexicon creation, empathy, distress 1. Introduction Deep learning, applied to ever larger datasets, has led to large improvements in performance in sentiment and emo- tion analysis. In light of this development, lexica, lists of words and associated weights for a particular affective vari- able, which used to be a key component for feature extrac- tion (Mohammad and Bravo-Marquez, 2017a), may seem obsolete. However, this is far from the truth. Lexica can be used as features to improve performance for sentence-level emotion prediction even in advanced neu- ral architectures (Mohammad and Bravo-Marquez, 2017b; De Bruyne et al., 2019). Word ratings are also often used to refine pre-trained embedding models for specific tasks (Yu et al., 2017; Khosla et al., 2018). But much more im- portantly, word ratings are relatively cheap to acquire and have been found to be robust across domains and even lan- guages, regarding their translational equivalents (Leveau et al., 2012; Warriner et al., 2013). This gives lexica a piv- otal role for processing under-resourced languages. Per- haps most importantly, using lexica allows for interpretable models since the resulting document-level predictions can be easily broken down to the words within it. This gives lexica an important role for building justifiable AI and ad- dressing related ethical challenges (Clos et al., 2017). In- terpretability is also crucial for NLP use in other academic disciplines such as psychology, social science, discourse linguistics, and the digital humanities, where understand- ing the nature of “constructs” (as psychologists call them) such as emotions is far more important than making accu- * These authors contributed equally to this work. Sven Buechel designed the MLFFN model (Section 3.). Jo˜ao Sedoc, along with …

Content truncated to 500 words