Neural Temporality Adaptation for Document Classification: Diachronic Word Embeddings and Domain Adaptation Models

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4113–4123 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4113 Neural Temporality Adaptation for Document Classification: Diachronic Word Embeddings and Domain Adaptation Models Xiaolei Huang and Michael J. Paul Department of Information Science University of Colorado, Boulder, CO 80309, USA {xiaolei.huang, mpaul}@colorado.edu Abstract Language usage can change across periods of time, but document classifiers models are usu- ally trained and tested on corpora spanning multiple years without considering temporal variations. This paper describes two com- plementary ways to adapt classifiers to shifts across time. First, we show that diachronic word embeddings, which were originally de- veloped to study language change, can also improve document classification, and we show a simple method for constructing this type of embedding. Second, we propose a time-driven neural classification model inspired by meth- ods for domain adaptation. Experiments on six corpora show how these methods can make classifiers more robust over time. 1 Introduction Language changes and varies over time, which can cause a degradation of performance in natural lan- guage processing models over time. For exam- ple, document classifiers are typically trained on historical data and tested on future data, where the performance tends to be worse. Recent re- search has shown that document classifiers can become more stable over time when trained in ways that specifically account for temporal varia- tions (Huang and Paul, 2018; He et al., 2018). We refer to this task of accounting for such variations during training as temporality adaptation. This paper investigates temporality adaptation in two ways. First, we explore how diachronic word embeddings, which encode time-varying representations of words, can be used in this set- ting. Recent research has used diachronic word embeddings to study how language changes over time (Kulkarni et al., 2015; Hamilton et al., 2016; Kutuzov et al., 2018). These studies have shown that shifts in the corpora across time cause changes in word contexts and consequently, changes in the learned representations. In our study, we further examine these shifts as they relate to important features for document classification. While other research has applied diachronic word embeddings to semantic change detection and validation (Mihalcea and Nastase, 2012; Kim et al., 2014; Kulkarni et al., 2015; Hamilton et al., 2016; Dubossarsky et al., 2017; Yao et al., 2018; Rudolph and Blei, 2018; Rosen- feld and Erk, 2018) and semantic relation anal- ysis (Liao and Cheng, 2016; Szymanski, 2017; Rosin et al., 2017), these types of embeddings have not been studied particularly for the docu- ment classification task. We show that neural clas- sifiers which use these embeddings can perform better on future data. As part of this work, we propose a new method for constructing diachronic words embeddings, which we show to be compet- itive with prior approaches. Second, we propose a neural classification model that adapts to changes in time using ideas from domain adaptation. We previously showed that out-of-the-box domain adaptation techniques can make n-gram …

Content truncated to 500 words