Experimenting with Distant Supervision for Emotion Classification

Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 482–491, Avignon, France, April 23 - 27 2012. c⃝2012 Association for Computational Linguistics Experimenting with Distant Supervision for Emotion Classification Matthew Purver∗and Stuart Battersby† ∗Interaction Media and Communication Group †Chatterbox Analytics School of Electronic Engineering and Computer Science Queen Mary University of London Mile End Road, London E1 4NS, UK m.purver@qmul.ac.uk stuart@cbanalytics.co.uk Abstract We describe a set of experiments using au- tomatically labelled data to train supervised classifiers for multi-class emotion detection in Twitter messages with no manual inter- vention. By cross-validating between mod- els trained on different labellings for the same six basic emotion classes, and testing on manually labelled data, we conclude that the method is suitable for some emotions (happiness, sadness and anger) but less able to distinguish others; and that different la- belling conventions are more suitable for some emotions than others. 1 Introduction We present a set of experiments into classify- ing Twitter messages into the six basic emotion classes of (Ekman, 1972). The motivation behind this work is twofold: firstly, to investigate the pos- sibility of detecting emotions of multiple classes (rather than purely positive or negative sentiment) in such short texts; and secondly, to investigate the use of distant supervision to quickly bootstrap large datasets and classifiers without the need for manual annotation. Text classification according to emotion and sentiment is a well-established research area. In this and other areas of text analysis and classifica- tion, recent years have seen a rise in use of data from online sources and social media, as these provide very large, often freely available datasets (see e.g. (Eisenstein et al., 2010; Go et al., 2009; Pak and Paroubek, 2010) amongst many others). However, one of the challenges this poses is that of data annotation: given very large amounts of data, often consisting of very short texts, written in unconventional style and without accompany- ing metadata, audio/video signals or access to the author for disambiguation, how can we easily pro- duce a gold-standard labelling for training and/or for evaluation and test? One possible solution that is becoming popular is crowd-sourcing the la- belling task, as the easy access to very large num- bers of annotators provided by tools such as Ama- zon’s Mechanical Turk can help with the problem of dataset size; however, this has its own attendant problems of annotator reliability (see e.g. (Hsueh et al., 2009)), and cannot directly help with the in- herent problem of ambiguity – using many anno- tators does not guarantee that they can understand or correctly assign the author’s intended interpre- tation or emotional state. In this paper, we investigate a different ap- proach via distant supervision (see e.g. (Mintz et al., 2009)). By using conventional markers of emotional content within the texts themselves as a surrogate for explicit labels, we can quickly re- trieve large subsets of (noisily) labelled data. This approach has the advantage of giving us direct access to the authors’ own …

Content truncated to 500 words