Recognizing Rare Social Phenomena in Conversation: Empowerment Detection in Support Group Chatrooms

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 104–113, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Recognizing Rare Social Phenomena in Conversation: Empowerment Detection in Support Group Chatrooms Elijah Mayfield, David Adamson, and Carolyn Penstein Ros´e Language Technologies Institute Carnegie Mellon University 5000 Forbes Ave, Pittsburgh, PA 15213 {emayfiel, dadamson, cprose}@cs.cmu.edu Abstract Automated annotation of social behavior in conversation is necessary for large-scale analysis of real-world conversational data. Important behavioral categories, though, are often sparse and often appear only in specific subsections of a conversation. This makes supervised machine learning difficult, through a combination of noisy features and unbalanced class distribu- tions. We propose within-instance con- tent selection, using cue features to selec- tively suppress sections of text and bias- ing the remaining representation towards minority classes. We show the effective- ness of this technique in automated anno- tation of empowerment language in online support group chatrooms. Our technique is significantly more accurate than multi- ple baselines, especially when prioritizing high precision. 1 Introduction Quantitative social science research has experi- enced a recent expansion, out of controlled set- tings and into natural environments. With this influx of interest comes new methodology, and the inevitable question arises of how to move towards testable hypotheses, using these uncon- trolled sources of data as scientific lenses into the real world. The study of conversational transcripts is a key domain in this new frontier. There are certain social and behavioral phenomena in conversation that cannot be easily identified through question- naire data, self-reported surveys, or easily ex- tracted user metadata. Examples of these social phenomena in conversation include overt displays of power (Prabhakaran et al., 2012) or indicators of rapport and relationship building (Wang et al., 2012). Manually annotating these social phenom- ena cannot scale to large data, so researchers turn to automated annotation of transcripts (Ros´e et al., 2008). While machine learning is highly effec- tive for annotation tasks with relatively balanced labels, such as sentiment analysis (Pang and Lee, 2004), more complex social functions are often rarer. This leads to unbalanced class label distri- butions and a much more difficult machine learn- ing task. Moreover, features indicative of rare so- cial annotations tend to be drowned out in favor of features biased towards the majority class. The net effect is that classification algorithms tend to bias towards the majority class, giving low accuracy for rare class detection. Automated annotation of social phenomena also brings opportunities for real-world applications. For example, real-time annotation of conversation can power adaptive intervention in collaborative learning settings (Rummel et al., 2008; Adamson and Ros´e, 2012). However, with the considerable power of automation comes great responsibility. It is critical to avoid intervening in the case of er- roneous annotations, as providing unnecessary or inappropriate support in such a setting has been shown to be harmful to group performance and so- cial cohesion (Dillenbourg, 2002; Stahl, 2012). We propose adaptations to existing machine learning algorithms which improve recognition of rare annotations …

Content truncated to 500 words