A Comprehensive Analysis of Preprocessing for Word Representation Learning in Affective Tasks

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5799–5810 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5799 A Comprehensive Analysis of Preprocessing for Word Representation Learning in Affective Tasks Nastaran Babanejad, Ameeta Agrawal, Aijun An, Manos Papagelis Department of Electrical Engineering and Computer Science, York University, Toronto, Canada {nasba, ameeta, aan, papaggel}@eecs.yorku.ca Abstract Affective tasks such as sentiment analysis, emotion classification and sarcasm detection have been popular in recent years due to abun- dance of user-generated data, accurate com- putational linguistic models, and broad range of relevant applications in various domains. At the same time, many studies have high- lighted the importance of text preprocessing, as an integral step to any natural language processing prediction model and downstream task. While preprocessing in affective systems is well-studied, preprocessing in word vector based models applied to affective systems, is not. To address this limitation, we conduct a comprehensive analysis of the role of prepro- cessing techniques in affective analysis based on word vector models. Our analysis is the first of its kind and provides useful insights of the importance of each preprocessing tech- nique when applied at the training phase, com- monly ignored in pretrained word vector mod- els, and/or at the downstream task phase. 1 Introduction Affective tasks such as sentiment analysis, emo- tion classification and sarcasm detection have en- joyed great popularity in recent years. This success can be largely attributed to the fundamental and straightforward nature of the methods employed, the availability of vast amounts of user-generated natural language data, and the wide range of useful applications, spanning from hate speech detection to monitoring the sentiment of financial markets and news recommendation (Djuric et al., 2015; Ba- banejad et al., 2019). Most early models of affect analysis employed pretrained word embeddings that have been obtained under the assumption of the distributional hypothesis (Mikolov et al., 2013; Devlin et al., 2018). The distributional hypothesis suggests that two words occurring frequently in similar linguistic contexts tend to be more semanti- cally similar, and therefore should be represented closer to one another in the embedding space. How- ever, while such embeddings are useful for several natural language processing (NLP) downstream tasks, they are known to be less suitable for affec- tive tasks in particular (Tang et al., 2014; Agrawal et al., 2018). Although some authors claim that there is a need for post-processing word embed- dings for affective tasks, others find that off-the- shelf vectors are very powerful for affective lexicon learning (Lison and Kutuzov, 2017). For example, word2vec (Mikolov et al., 2013) estimates the pair of words ‘happy’ and ‘sad’ to be more similar than the pair of words ‘happy’ and ‘joy’, which is counterintuitive, and might affect the accuracy performance of the models that depend on it. To address the limitations of traditional word em- beddings, several techniques have been proposed, including task-specific fine-tuning (Devlin et al., 2018), retrofitting (Faruqui et al., 2014), represent- ing emotion with vectors using a multi-task training …

Content truncated to 500 words