Dynamic Language Models for Streaming Text

Dynamic Language Models for Streaming Text Dani Yogatama∗ Chong Wang∗ Bryan R. Routledge† Noah A. Smith∗ Eric P. Xing∗ ∗School of Computer Science †Tepper School of Business Carnegie Mellon University Pittsburgh, PA 15213, USA ∗{dyogatama,chongw,nasmith,epxing}@cs.cmu.edu, †routledge@cmu.edu Abstract We present a probabilistic language model that captures temporal dynamics and conditions on arbitrary non-linguistic context features. These context features serve as important indicators of language changes that are otherwise difficult to capture using text data by itself. We learn our model in an efficient online fashion that is scalable for large, streaming data. With five streaming datasets from two different genres— economics news articles and social media—we evaluate our model on the task of sequential language modeling. Our model consistently outperforms competing models. 1 Introduction Language models are a key component in many NLP applications, such as machine translation and ex- ploratory corpus analysis. Language models are typi- cally assumed to be static—the word-given-context distributions do not change over time. Examples include n-gram models (Jelinek, 1997) and proba- bilistic topic models like latent Dirichlet allocation (Blei et al., 2003); we use the term “language model” to refer broadly to probabilistic models of text. Recently, streaming datasets (e.g., social media) have attracted much interest in NLP. Since such data evolve rapidly based on events in the real world, as- suming a static language model becomes unrealistic. In general, more data is seen as better, but treating all past data equally runs the risk of distracting a model with irrelevant evidence. On the other hand, cau- tiously using only the most recent data risks overfit- ting to short-term trends and missing important time- insensitive effects (Blei and Lafferty, 2006; Wang et al., 2008). Therefore, in this paper, we take steps toward methods for capturing long-range temporal dynamics in language use. Our model also exploits observable context vari- ables to capture temporal variation that is otherwise difficult to capture using only text. Specifically for the applications we consider, we use stock market data as exogenous evidence on which the language model depends. For example, when an important company’s price moves suddenly, the language model should be based not on the very recent history, but should be similar to the language model for a day when a similar change happened, since people are likely to say similar things (either about that com- pany, or about conditions relevant to the change). Non-linguistic contexts such as stock price changes provide useful auxiliary information that might indi- cate the similarity of language models across differ- ent timesteps. We also turn to a fully online learning framework (Cesa-Bianchi and Lugosi, 2006) to deal with non- stationarity and dynamics in the data that necessitate adaptation of the model to data in real time. In on- line learning, streaming examples are processed only when they arrive. Online learning also eliminates the need to store large amounts of data in memory. Strictly speaking, online learning is distinct from stochastic learning, which for language models built on massive datasets has been explored by Hoffman …

Content truncated to 500 words