Deep Temporal-Recurrent-Replicated-Softmax for Topical Trends over Time

Proceedings of NAACL-HLT 2018, pages 1079–1089 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Deep Temporal-Recurrent-Replicated-Softmax for Topical Trends over Time Pankaj Gupta1,2, Subburam Rajaram1, Hinrich Sch¨utze2, Bernt Andrassy1 1Corporate Technology, Machine-Intelligence (MIC-DE), Siemens AG Munich, Germany 2CIS, University of Munich (LMU) Munich, Germany {pankaj.gupta, subburam.rajaram, bernt.andrassy}@siemens.com pankaj.gupta@campus.lmu.de | inquiries@cislmu.org Abstract Dynamic topic modeling facilitates the iden- tification of topical trends over time in tem- poral collections of unstructured documents. We introduce a novel unsupervised neural dy- namic topic model named as Recurrent Neural Network-Replicated Softmax Model (RNN- RSM), where the discovered topics at each time influence the topic discovery in the sub- sequent time steps. We account for the tempo- ral ordering of documents by explicitly mod- eling a joint distribution of latent topical de- pendencies over time, using distributional es- timators with temporal recurrent connections. Applying RNN-RSM to 19 years of articles on NLP research, we demonstrate that com- pared to state-of-the art topic models, RNN- RSM shows better generalization, topic inter- pretation, evolution and trends. We also intro- duce a metric (named as SPAN) to quantify the capability of dynamic topic model to capture word evolution in topics over time. 1 Introduction Topic Detection and Tracking (Allan et al., 1998) is an important area of natural language process- ing to find topically related ideas that evolve over time in a sequence of text collections and exhibit temporal relationships. The temporal aspects of these collections can present valuable insight into the topical structure of the collections and can be quantified by modeling the dynamics of the under- lying topics discovered over time. Problem Statement: We aim to generate tem- poral topical trends or automatic overview time- lines of topics for a time sequence collection of documents. This involves the following three tasks in dynamic topic analysis: (1) Topic Structure De- tection (TSD): Identifying main topics in the doc- ument collection. (2) Topic Evolution Detection (TED): Detecting the emergence of a new topic Language Model Distributional Semantics Word Vectors Linear Model Rule Set Neural Language Model Word Embedding Semantic Representation Distributional models Neural Network 1996 2014 ... h(0) h(t-1) RSM h(t) RSM h(T) RSM ... V(t-1) V(t) V(T) (t-1) (t) (T) Figure 1: (Left): Word Usage over time for Topic (Word Representation) in scholarly articles. (Right): RSM-based dynamic topic model with explicit tempo- ral topic dependence and recognizing how it grows or decays over time (Allan, 2002). (3) Temporal Topic Characteriza- tion (TTC): Identifying the characteristics for each of the main topics in order to track the words’ us- age (keyword trends) for a topic over time i.e. topi- cal trend analysis for word evolution (Fig 1, Left). Probabilistic static topic models, such as La- tent Dirichlet Allocation (LDA) (Blei et al., 2003) and its variants (Wang and McCallum, 2006; Hall et al., 2008; Gollapalli and Li, 2015) have been investigated to examine the emergence of top- ics from historical documents. Another vari- ant known as Replicated Softmax (RSM) (Hinton and Salakhutdinov, 2009) has …

Content truncated to 500 words