Temporally-Informed Analysis of Named Entity Recognition

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7605–7617 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7605 Temporally-Informed Analysis of Named Entity Recognition Shruti Rijhwani∗ Language Technologies Institute Carnegie Mellon University srijhwan@cs.cmu.edu Daniel Preot¸iuc-Pietro Bloomberg dpreotiucpie@bloomberg.net Abstract Natural language processing models often have to make predictions on text data that evolves over time as a result of changes in language use or the information described in the text. However, evaluation results on ex- isting data sets are seldom reported by taking the timestamp of the document into account. We analyze and propose methods that make better use of temporally-diverse training data, with a focus on the task of named entity recog- nition. To support these experiments, we in- troduce a novel data set of English tweets an- notated with named entities.1 We empirically demonstrate the effect of temporal drift on per- formance, and how the temporal information of documents can be used to obtain better mod- els compared to those that disregard temporal information. Our analysis gives insights into why this information is useful, in the hope of informing potential avenues of improvement for named entity recognition as well as other NLP tasks under similar experimental setups. 1 Introduction Natural language processing models are now de- ployed on a large scale in many applications and used to drive automatic analyses or for making pre- dictions. The usual setup is that these models are trained and evaluated on the data available at model building time, but are used to make inferences on data coming in at a future time, making models sus- ceptible to data drift. The data distribution of the test set used to measure the model’s performance after training may be different from the distribution of data from future time periods (Huang and Paul, 2018). This temporal drift in data often results in lower performance during inference. Drift is es- pecially prevalent in information extraction tasks, ∗Work done during an internship at Bloomberg 1Our data set is available at https://github.com/ shrutirij/temporal-twitter-corpus. such as named entity recognition (NER), where the context and the target entities differ across time as a result of changes in language use or the events being discussed (Derczynski et al., 2016). Despite its intuitive value, there has been little research on using the temporal information con- tained in text documents to inform modeling of a task (Huang and Paul, 2018; He et al., 2018), and no past research on modeling sequence label- ing tasks in particular. Since sequence labeling models are currently trained and evaluated by ran- domly splitting the available data, performance is measured in an artificially temporal drift-free sce- nario that is not realistic or similar to how models are used in practice (Dredze et al., 2010). When splitting the available training data temporally and testing on the data from the most recent time period, we formulate the following hypotheses: a) models trained on data from a closer time to the test set obtain better results, …

Content truncated to 500 words