Stylometric Analysis of Scientific Articles

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 327–337, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Stylometric Analysis of Scientific Articles Shane Bergsma, Matt Post, David Yarowsky Department of Computer Science and Human Language Technology Center of Excellence Johns Hopkins University Baltimore, MD 21218, USA sbergsma@jhu.edu, post@cs.jhu.edu, yarowsky@cs.jhu.edu Abstract We present an approach to automatically re- cover hidden attributes of scientific articles, such as whether the author is a native English speaker, whether the author is a male or a fe- male, and whether the paper was published in a conference or workshop proceedings. We train classifiers to predict these attributes in computational linguistics papers. The classi- fiers perform well in this challenging domain, identifying non-native writing with 95% accu- racy (over a baseline of 67%). We show the benefits of using syntactic features in stylom- etry; syntax leads to significant improvements over bag-of-words models on all three tasks, achieving 10% to 25% relative error reduction. We give a detailed analysis of which words and syntax most predict a particular attribute, and we show a strong correlation between our predictions and a paper’s number of citations. 1 Introduction Stylometry aims to recover useful attributes of doc- uments from the style of the writing. In some do- mains, statistical techniques have successfully de- duced author identity (Mosteller and Wallace, 1984), gender (Koppel et al., 2003), native language (Kop- pel et al., 2005), and even whether an author has de- mentia (Le et al., 2011). Stylometric analysis is im- portant to marketers, analysts and social scientists because it provides demographic data directly from raw text. There has been growing interest in apply- ing stylometry to the content generated by users of Internet applications, e.g., detecting author ethnic- ity in social media (Eisenstein et al., 2011; Rao et Native/Non-native Male/Female Conference/Workshop Stylometric Analysis of Scientific Articles Abstract We present an approach to automatically re- cover hidden attributes of scientific articles, such as whether the author is a native En- glish speaker. We train classifiers to predict these attributes in computational linguistics papers. The classifiers perform well in this challenging domain, identifying non-native writing with 95% accuracy (over a baseline of 67%), and outperforming reasonable base- lines on two other difficult tasks. We show the benefits of using syntactic features in stylom- etry; syntax leads to significant improvements over bag-of-words models on all three tasks, achieving 10% to 25% relative error reduction. We give an insightful analysis of which words and syntax most predict a particular attribute, and we show a strong correlation between our predictions and a paper’s number of citations. 1 Introduction Stylometry aims to recover useful attributes of doc- uments from the style of their writing. In some domains, statistical techniques have successfully deduced author identities (Mosteller and Wallace, 1984), gender (Koppel et al., 2003), native language (Koppel et al., 2005), and even whether an author has dementia (Le et al., 2011). Stylometric analysis is important to marketers, analysts …

Content truncated to 500 words