Emotion Enriched Retrofitted Word Embeddings

Proceedings of the 29th International Conference on Computational Linguistics, pages 4136–4148 October 12–17, 2022. 4136 Emotion Enriched Retrofitted Word Embeddings Sapan Shah1,2, Sreedhar Reddy1, and Pushpak Bhattacharyya2 1TCS Research, Tata Consultancy Services, Pune 2Indian Institute of Technology Bombay, Mumbai {sapan.hs,sreedhar.reddy}@tcs.com pb@cse.iitb.ac.in Abstract Word embeddings learned using the distribu- tional hypothesis (e.g., GloVe, Word2vec) are good at encoding various lexical-semantic re- lations. However, they do not capture the emo- tion aspects of words. We present a novel retrofitting method for updating the vectors of emotion bearing words like fun, offence, an- gry, etc. The retrofitted embeddings achieve better inter-cluster and intra-cluster distance for words having the same emotions, e.g., the joy cluster containing words like fun, happi- ness, etc., and the anger cluster with words like offence, rage, etc., as evaluated through different cluster quality metrics. For the down- stream tasks on sentiment analysis and sar- casm detection, simple classification models, such as SVM and Attention Net, learned using our retrofitted embeddings perform better than their pre-trained counterparts (about 1.5% im- provement in F1-score) as well as other bench- marks. Furthermore, the difference in perfor- mance is more pronounced in the limited data setting. 1 Introduction Word embedding models inspired from the distri- butional hypothesis (Harris, 1954) have one major limitation: they mix semantic similarity with other types of semantic relatedness (Hill et al., 2015). For instance, consider cheap and expensive. Though opposite in meaning, the distributional vectors of these words are similar since they occur in nearly identical contexts. This is problematic for many applications such as text simplification, dialogue state tracking, etc. To address this, researchers have proposed various models that leverage knowledge resources to improve word embeddings. At a high level, these models are categorized into two types: Joint specialization models (Yu and Dredze, 2014; Liu et al., 2015); and Retrofitting (post-processing) models (Faruqui et al., 2015; Mrkši´c et al., 2016). Joint specialization models typically modify the word pair GloVe RETripletGBal (angry, offence) 0.2339 0.3924 (angry, enjoy) 0.3400 0.2950 (fun, closeness) 0.2232 0.3688 (fun, miserable) 0.3105 0.2812 Table 1: Cosine similarity between words from same and different emotion categories: pre-trained GloVe vs. embeddings retrofitted by our method RETripletGBal optimization objective of distributional models by integrating external knowledge into the objective function. In contrast, retrofitting models first gener- ate training data from knowledge resources in the form of constraints and then modify the pre-trained embeddings in a post-processing step so that they respect the constraints. These approaches focus mainly on constraints from relations such as syn- onymy, antonymy, hypernymy, etc., that are present in WordNet, Paraphrase database, etc. While pre-trained embeddings and their retrofitted versions are good at encoding various lexico-semantic relations, they do not consider the emotion content of words. For example, consider words such as angry and offence that evoke anger emotion and words such as fun and enjoy eliciting joy. Table 1 shows cosine similarity as computed using pre-trained GloVe embeddings. Even though angry and offence evoke the same emotion (anger), their cosine similarity is lower …

Content truncated to 500 words