Demographic Factors Improve Classification Performance

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 752–762, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Demographic Factors Improve Classification Performance Dirk Hovy Center for Language Technology University of Copenhagen, Denmark Njalsgade 140 dirk@cst.dk Abstract Extra-linguistic factors influence language use, and are accounted for by speakers and listeners. Most natural language pro- cessing (NLP) tasks to date, however, treat language as uniform. This assump- tion can harm performance. We investi- gate the effect of including demographic information on performance in a variety of text-classification tasks. We find that by including age or gender information, we consistently and significantly improve performance over demographic-agnostic models. These results hold across three text-classification tasks in five languages. 1 Introduction When we use language, we take demographic factors of the speakers into account. In other words, we do have certain expectations as to who uses “super cute,” “rather satisfying,” or “rad, dude.” Sociolinguistics has long since studied the interplay between demographic factors and lan- guage use (Labov, 1964; Milroy and Milroy, 1992; Holmes, 1997; Macaulay, 2001; Macaulay, 2002; Barbieri, 2008; Wieling et al., 2011; Rickford and Price, 2013, inter alia).1 These factors greatly in- fluence word choice, syntax, and even semantics. In natural language processing (NLP), however, we have largely ignored demographic factors, and treated language as a uniform medium. It was ir- relevant, (and thus not modeled) whether a text was produced by a middle-aged man, an elderly lady, or a teenager. These three groups, how- ever, differ along a whole host of demographic axes, and these differences are reflected in their language use. 1Apart from the demographic factors, other factors such as mood, interpersonal relationship, authority, language atti- tude, etc. contribute to our perception of language. A model that is agnostic to demographic dif- ferences will lose these distinctions, and perfor- mance suffers whenever the model is applied to a new demographic. Historically, the demograph- ics of training and test data (newswire) were rela- tively homogenous, language was relatively uni- form, and information the main objective. Un- der these uniform conditions, the impact of demo- graphics on performance was small. Lately, however, NLP is increasingly applied to other domains, such as social media, where language is less canonical, demographic informa- tion about the author is available, and the authors’ goals are no longer purely informational. The in- fluence of demographic factors in this medium is thus much stronger than on the data we have tra- ditionally used to induce models. The resulting performance drops have often been addressed via various domain adaptation approaches (Blitzer et al., 2006; Daume III and Marcu, 2006; Reichart and Rappoport, 2007; Chen et al., 2009; Daum´e et al., 2010; Chen et al., 2011; Plank and Moschitti, 2013; Plank et al., 2014; Hovy et al., 2015b, inter alia). However, the authors and target demograph- ics of social media differ radically from those in newswire text, and domain might in some case …

Content truncated to 500 words