Emotion Corpus Construction Based on Selection from Hashtags

Emotion Corpus Construction Based on Selection from Hashtags Minglei Li, Yunfei Long, Qin Lu, Wenjie Li Department of Computing, The Hong Kong Polytechnic University Hung Hom, Kowloon, Hong Kong E-mail: {csmli, csylong, csluqin, cswjli}@comp.polyu.edu.hk Abstract The availability of labelled corpus is of great importance for supervised learning in emotion classification tasks. Because it is time-consuming to manually label text, hashtags have been used as naturally annotated labels to obtain a large amount of labelled training data from microblog. However, natural hashtags contain too much noise for it to be used directly in learning algorithms. In this paper, we design a three-stage semi-automatic method to construct an emotion corpus from microblogs. Firstly, a lexicon based voting approach is used to verify the hashtag automatically. Secondly, a SVM based classifier is used to select the data whose natural labels are consistent with the predicted labels. Finally, the remaining data will be manually examined to filter out the noisy data. Out of about 48K filtered Chinese microblogs, 39k microblogs are selected to form the final corpus with the Kappa value reaching over 0.92 for the automatic parts and over 0.81 for the manual part. The proportion of automatic selection reaches 54.1%. Thus, the method can reduce about 44.5% of manual workload for acquiring quality data. Experiment on a classifier trained on this corpus shows that it achieves comparable results compared to the manually annotated NLP&CC2013 corpus. Keywords: hashtag; emotion corpus construction; high-quality label selection 1. Introduction and Related Works Emotion classification for text from the social media (such as Twitter, Sina Weibo) is becoming more and more important because emotions expressed in these media can have a social impact in the society. Many supervised learning methods have been employed to solve this problem. However, supervised methods require a large amount of labelled training data. Obtaining labelled data, if annotated manually, is very time-consuming especially for emotion analysis where training data can be quite skewed for multiple classes. There are a number of emotion corpora from previous research works. For the English language, SemEval2007 (Strapparava and Mihalcea, 2007) consists of only 1,250 news headlines labelled with the six Ekman emotion labels. The ISEAR dataset (Scherer and Wallbott, 1994) consists of 7,666 sentences generated by participants through questionnaires. The Affect Dataset (Alm 2009) consists of more than 15,000 sentences from fairy tales with five emotion labels. For Chinese, the Ren-CECps (Quan and Ren, 2010) emotion corpus consists of 1,487 documents and 35,096 sentences from web blogs with eight emotions. The limitations of the above datasets are that they are either too small in size or not proper for social media text analysis. Another social media orientated Chinese corpus NLP&&CC 2013 (Yuanlin et al., 2014) consists of 14,000 microblogs and 45,431 sentences from microblogs with 8 labels (including “none” label, meaning no emotion) through manual annotation only 7,300 microblogs contain emotions and the size is still quite small. Those corpora are not suitable for large-scale emotion analysis. Many research studies employ distant supervision and they take …

Content truncated to 500 words