Modeling Diachronic Change in Scientific Writing with Information Density

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 750–761, Osaka, Japan, December 11-17 2016. Modeling Diachronic Change in Scientific Writing with Information Density Raphael Rubino†‡ Stefania Degaetano-Ortlieb† †Universit¨at des Saarlandes, ‡DFKI Saarbr¨ucken, Germany {s.degaetano,e.teich}@mx.uni-saarland.de {raphael.rubino,josef.vangenabith}@uni-saarland.de Elke Teich† Josef van Genabith†‡ Abstract Previous linguistic research on scientific writing has shown that language use in the scientific domain varies considerably in register and style over time. In this paper we investigate the in- troduction of information theory inspired features to study long term diachronic change on three levels: lexis, part-of-speech and syntax. Our approach is based on distinguishing between sen- tences from 19th and 20th century scientific abstracts using supervised classification models. To the best of our knowledge, the introduction of information theoretic features to this task is novel. We show that these features outperform more traditional features, such as token or character n-grams, while leading to more compact models. We present a detailed analysis of feature in- formativeness in order to gain a better understanding of diachronic change on different linguistic levels. 1 Introduction Supervised classification has been applied to various natural language processing tasks over the past decades. To date, however, distinguishing between time periods has not received extensive attention. Early research on classifying time periods is presented in de Jong et al. (2005) for Dutch. Dalli and Wilks (2006) and Kumar et al. (2011) use word frequencies for temporal classification of documents, while Sagi et al. (2009) and Kim et al. (2014) predict semantic changes over time. While lexical features are commonly used for classification approaches of time periods, features based on more abstract linguistic levels have not yet been widely investigated. In our study, we use supervised classification to distinguish scientific abstracts written in the 19th and 20th century at the sentence-level. From previous work, we know that in the scientific domain, shared expertise among authors and audience affects their language use. Over a longer time period, it drives the evolution of domain-specific language with respect to lexis (Halliday, 1988; Teich et al., 2016) and a more standardized and convention-driven style with respect to grammar (Biber and Gray, 2011; Biber and Gray, 2016; Banks, 2005). Considering that language variation affects all linguistic levels — from sounds and words to syntactic structure — we investigate a set of features extracted at the lexis, part-of-speech and syntactic levels to test how well they act as predictors of time period-specific language use. Moreover, based on psycholin- guistic evidence it has been shown that language users choose those linguistic options that they know to be relatively predictable in a specific context to optimize communication (Hale, 2001; Levy, 2008; Demberg and Keller, 2008). To model communication in this sense, in our research we employ features based on the information-theoretic notion of surprisal or information density. Specifically, we make use of information theory inspired features on the linguistic levels of lexis, part-of-speech and syntax. In addition, these features allow an unlexicalized dense-vector representa- tion, which enormously reduces the …

Content truncated to 500 words