That’s So Annoying!!!: A Lexical and Frame-Semantic Embedding Based Data Augmentation Approach to Automatic Categorization of Annoying Behaviors using #petpeeve Tweets

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2557–2563, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. That’s So Annoying!!!: A Lexical and Frame-Semantic Embedding Based Data Augmentation Approach to Automatic Categorization of Annoying Behaviors using #petpeeve Tweets ∗ William Yang Wang and Diyi Yang Language Technologies Institute School of Computer Science Carnegie Mellon University {yww,diyiy}@cs.cmu.edu Abstract We propose a novel data augmentation ap- proach to enhance computational behav- ioral analysis using social media text. In particular, we collect a Twitter corpus of the descriptions of annoying behaviors us- ing the #petpeeve hashtags. In the qual- itative analysis, we study the language use in these tweets, with a special focus on the fine-grained categories and the ge- ographic variation of the language. In quantitative analysis, we show that lexi- cal and syntactic features are useful for au- tomatic categorization of annoying behav- iors, and frame-semantic features further boost the performance; that leveraging large lexical embeddings to create addi- tional training instances significantly im- proves the lexical model; and incorporat- ing frame-semantic embedding achieves the best overall performance. 1 Introduction In the ever-expanding era of social media, many scientific disciplines, such as health and health- care, biology, and learning sciences, have adopted computational approaches to exploit patterns and behaviors in large datasets (Wang et al., 2015; Chen and Lonardi, 2009; Baker and Yacef, 2009). In contrast, the primary methods for behavioral sciences still rely on lab experiments with limited amount of subjects, which are time consuming and financially expensive. In addition to this, it is also difficult to obtain a set of samples with geograph- ∗We understand that many people find long titles annoy- ing, so we intentionally use a very long one to help people understand what “pet peeve” means. Figure 1: An anonymized example of #petpeeve tweets. ical variations in traditional lab-based behavioral experiments. While the social media data are abundantly available, computational approaches to behavioral sciences using Twitter are not well-studied. Even when statistical techniques are applied to these tasks, their concentration has been on simple sta- tistical significance tests and descriptive statis- tics (De Charms, 2013; Zhang et al., 2013). There- fore, we believe that statistical natural language processing techniques are needed for insightful analysis and interpretation in behavioral studies. In this paper, we use Twitter as a corpus for computational behavioral science. More specifi- cally, we focus on a case study of analyzing an- noying behaviors. To do this, we exploit a corpus of 9 million tweets (Cheng et al., 2010), and ex- tract the tweets that describe these behaviors us- ing the #petpeeve hashtags. #petpeeve is a pop- ular Twitter hashtag, which describes behaviors that might be annoying to others. An example of #petpeeve tweets is shown in Figure 1. To fa- cilitate the analysis, we manually annotate 3,375 tweets with 60 fine-grained categories, which will be described in Section 3. We use a sparse mixed- effects topic model to analyze the salient words in each category, as …

Content truncated to 500 words