Predicting a Scientific Community’s Response to an Article

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 594–604, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Predicting a Scientific Community’s Response to an Article Dani Yogatama Michael Heilman Brendan O’Connor Chris Dyer School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213, USA {dyogatama,mheilman,brenocon,cdyer}@cs.cmu.edu Bryan R. Routledge Tepper School of Business Carnegie Mellon University Pittsburgh, PA 15213, USA routledge@cmu.edu Noah A. Smith School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213, USA nasmith@cs.cmu.edu Abstract We consider the problem of predicting mea- surable responses to scientific articles based primarily on their text content. Specif- ically, we consider papers in two fields (economics and computational linguistics) and make predictions about downloads and within-community citations. Our approach is based on generalized linear models, allowing interpretability; a novel extension that cap- tures first-order temporal effects is also pre- sented. We demonstrate that text features significantly improve accuracy of predictions over metadata features like authors, topical categories, and publication venues. 1 Introduction Written communication is an essential component of the complex social phenomenon of science. As such, natural language processing is well-positioned to provide tools for understanding the scientific pro- cess, by analyzing the textual artifacts (papers, pro- ceedings, etc.) that it produces. This paper is about modeling collections of scientific documents to un- derstand how their textual content relates to how a scientific community responds to them. While past work has often focused on citation structure (Borner et al., 2003; Qazvinian and Radev, 2008), our em- phasis is on the text content, following Ramage et al. (2010) and Gerrish and Blei (2010). Instead of task-independent exploratory data anal- ysis (e.g., topic modeling) or multi-document sum- marization, we consider supervised models of the collective response of a scientific community to a published article. There are many measures of im- pact of a scientific paper; ours come from direct measurements of the number of downloads (from an established website where prominent economists post papers before formal publication) and citations (within a fixed scientific community). We adopt a discriminative approach based on generalized lin- ear models that can make use of any text or meta- data features, and show that simple lexical fea- tures offer substantial power in modeling out-of- sample response and in forecasting response for fu- ture articles. Realistic forecasting evaluations re- quire methodological care beyond the usual best practices of train/test separation, and we elucidate these issues. In addition, we introduce a new regularization technique that leverages the intuition that the rela- tionship between observable features and response should evolve smoothly over time. This regularizer allows the learner to rely more strongly on more re- cent evidence, while taking into account a long his- tory of training data. Our time series-inspired regu- larizer is computationally efficient in learning and is a significant advance over earlier text-driven fore- casting models that ignore the time variable alto- gether (Kogan et al., 2009; Joshi et al., 2010). We evaluate our approaches in two novel experi- mental settings: …

Content truncated to 500 words