Temporal Effects on Pre-trained Models for Language Processing Tasks

Temporal Effects on Pre-trained Models for Language Processing Tasks Oshin Agarwal University of Pennsylvania, USA oagarwal@seas.upenn.edu Ani Nenkova Adobe Research, USA nenkova@adobe.com Abstract Keeping the performance of language tech- nologies optimal as time passes is of great practical interest. We study temporal effects on model performance on downstream language tasks, establishing a nuanced terminology for such discussion and identifying factors essen- tial to conduct a robust study. We present experiments for several tasks in English where the label correctness is not dependent on time and demonstrate the importance of distinguish- ing between temporal model deterioration and temporal domain adaptation for systems us- ing pre-trained representations. We find that, depending on the task, temporal model deteri- oration is not necessarily a concern. Temporal domain adaptation, however, is beneficial in all cases, with better performance for a given time period possible when the system is trained on temporally more recent data. Therefore, we also examine the efficacy of two approaches for temporal domain adaptation without hu- man annotations on new data. Self-labeling shows consistent improvement and notably, for named entity recognition, leads to bet- ter temporal adaptation than even human annotations. 1 Introduction Language models capture properties of language, such as semantics of words and phrases and their typical usage, as well as facts about the world expressed in the language sample on which they were trained. Effective solutions for many lan- guage tasks depend, to a varying degree, on the background knowledge encoded in language mod- els. Performance may degrade as language and world-related facts change. In some scenarios, language will change as a result of deploying a system that uses the language to make a predic- tion, as in spam detection (Fawcett, 2003). But most change is not driven by such adversarial adaptations: the language expressing sentiment in product reviews (Lukes and Søgaard, 2018), the named entities and the contexts in which they are discussed on social media (Fromreide et al., 2014; Rijhwani and Preotiuc-Pietro, 2020), and language markers of political ideology (Huang and Paul, 2018) all change over time. Whether and how this change impacts the per- formance of different language technologies is a question of great practical interest. Yet research on quantifying how model performance changes with time has been sporadic. Moreover, approaches to solving language tasks have evolved rapidly, from bag of words models that rely on a small num- ber of fixed words represented as strings, without underlying meaning, to fixed dense word repre- sentations such as word2vec and GloVe (Mikolov et al., 2013; Pennington et al., 2014) and large contextualized representations of language (Peters et al., 2018; Devlin et al., 2019) that are trained on task-independent text to provide a backbone rep- resentation for word meaning. The swift change in approaches has made it hard to understand how representations and the data used to train them modulate the changes in system performance over time. We present experiments (§4 & §5) designed to study temporal effects on downstream language tasks, disentangling worsening model perfor- mance due …

Content truncated to 500 words