TILM: Neural Language Models with Evolving Topical Influence

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 778–788 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 778 TILM: a Neural Language Model with Evolving Topical Influence Shubhra Kanti Karmaker Santu and Kalyan Veeramachaneni MIT LIDS Cambridge, Massachusetts, USA santu@mit.edu kalyanv@mit.edu ChengXiang Zhai Department of Computer Science University of Illinois Urbana Champaign Urbana, Illinois, USA czhai@Illinois.edu Abstract Content of text data are often influenced by contextual factors which often evolve over time (e.g., content of social media are often in- fluenced by topics covered in the major news streams). Existing language models do not consider the influence of such related evolv- ing topics, and thus are not optimal. In this paper, we propose to incorporate such topical- influence into a language model to both im- prove its accuracy and enable cross-stream analysis of topical influences. Specifically, we propose a novel language model called Topical Influence Language Model (TILM), which is a novel extension of a neural language model to capture the influences on the contents in one text stream by the evolving topics in another related (or possibly same) text stream. Experi- mental results on six different text stream data comprised of conference paper titles show that the incorporation of evolving topical influence into a language model is beneficial and TILM outperforms multiple baselines in a challeng- ing task of text forecasting. In addition to serv- ing as a language model, TILM further enables interesting analysis of topical influence among multiple text streams. 1 Introduction and Motivation Language modeling plays a central role in many Natural Language Processing (NLP) tasks and ap- plications. Neural language models have attracted much attention recently due to their superior per- formance (Dieng et al., 2016; Kiros et al., 2014; Kiddon et al., 2016; Ranzato et al., 2015; Devlin et al., 2018; Peters et al., 2018). One common lim- itation of the existing neural language models is that they cannot model the potential influence of related contextual factors on text content genera- tion. However, as text data are produced by hu- mans based on their observations of the real world, the content of text data are generally influenced by many contextual factors, and thus, it is necessary to model the influence of those contextual factors on the generation of text content in order to opti- mize language modeling. For example, the content of social media may be influenced by the popular topics in news stream; another example is that the content of research papers in one research commu- nity such as Information Retrieval are often influ- enced by the topics of research papers published in the same community in the past or research papers published in another related research community such as Machine Learning since the general algo- rithms developed in the latter may be applied to solve application problems in the former. In this paper, we propose to incorporate such topical influence into a language model to both im- prove its accuracy and enable cross-stream anal- ysis of …

Content truncated to 500 words