Analyzing Biases in Human Perception of User Age and Gender from Text

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 843–854, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Analyzing Biases in Human Perception of User Age and Gender from Text Lucie Flekova∗ Ubiquitous Knowledge Processing Lab Department of Computer Science Technische Universit¨at Darmstadt www.ukp.tu-darmstadt.de Jordan Carpenter and Salvatore Giorgi Positive Psychology Center University of Pennsylvania Lyle Ungar and Daniel Preot¸iuc-Pietro Computer & Information Science University of Pennsylvania Abstract User traits disclosed through written text, such as age and gender, can be used to per- sonalize applications such as recommender systems or conversational agents. However, human perception of these traits is not per- fectly aligned with reality. In this paper, we conduct a large-scale crowdsourcing ex- periment on guessing age and gender from tweets. We systematically analyze the qual- ity and possible biases of these predictions. We identify the textual cues which lead to miss-assessments of traits or make annota- tors more or less confident in their choice. Our study demonstrates that differences be- tween real and perceived traits are notewor- thy and elucidates inaccurately used stereo- types in human perception. 1 Introduction There are notable differences between actual user traits and their perception by others (John and Robins, 1994; Kobrynowicz and Branscombe, 1997). Assessments of the perceived traits are de- pendent, for example, on the interpretation skills of a judge (Kenny and Albright, 1987) and the abil- ity of users to deliberately adjust their behavior to the way they intend to be perceived e.g., for fol- lowing a social goal (Kanellakos, 2002). People typically use stereotypes – a set of beliefs, gener- alizations, and associations about a social group – to make judgements about others. The discrepancy between stereotypes and actual group differences is ∗ Project carried out during a research stay at the Univer- sity of Pennsylvania an important topic in psychological research (Eagly, 1995; Dovidio et al., 1996; John and Robins, 1994; Kobrynowicz and Branscombe, 1997). Such differ- ences are likely reflected through one’s writing. With the Internet a substantial part of daily life, users leave enough footprints which allow algo- rithms to learn a range of individual traits, some with even higher accuracy than the users’ own fam- ily (Youyou et al., 2015). With an increase in read- ily available user generated content, prediction of user attributes has become more popular than ever. Researchers built learning models to infer different user traits from text, such as age (Rao et al., 2010), gender (Burger et al., 2011; Flekova and Gurevych, 2013), location (Eisenstein et al., 2010), political orientation (Volkova et al., 2014), income (Preot¸iuc- Pietro et al., 2015c), socio-economic status (Lam- pos et al., 2016), popularity (Lampos et al., 2014), personality (Schwartz et al., 2013) or mental ill- nesses (De Choudhury et al., 2013; Coppersmith et al., 2014; Preot¸iuc-Pietro et al., 2015a). Prediction models are trained on large data sets with labels extracted either from user self- reports (Preot¸iuc-Pietro et al., 2015b) or perceived from annotations (Volkova et al., 2015; Volkova and …

Content truncated to 500 words