Scaling Hidden Markov Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1341–1349, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1341 Scaling Hidden Markov Language Models Justin T. Chiu and Alexander M. Rush Department of Computer Science Cornell Tech {jtc257,arush}@cornell.edu Abstract The hidden Markov model (HMM) is a funda- mental tool for sequence modeling that cleanly separates the hidden state from the emission structure. However, this separation makes it difficult to fit HMMs to large datasets in mod- ern NLP, and they have fallen out of use due to very poor performance compared to fully observed models. This work revisits the chal- lenge of scaling HMMs to language modeling datasets, taking ideas from recent approaches to neural modeling. We propose methods for scaling HMMs to massive state spaces while maintaining efficient exact inference, a com- pact parameterization, and effective regulariza- tion. Experiments show that this approach leads to models that are more accurate than previous HMM and n-gram-based methods, making progress towards the performance of state-of-the-art neural models. 1 Introduction Hidden Markov models (HMMs) are a fundamen- tal latent-variable model for sequential data, with a rich history in NLP. They have been used exten- sively in tasks such as tagging (Merialdo, 1994), alignment (Vogel et al., 1996), and even, in a few cases, language modeling (Kuhn et al., 1994; Huang, 2011). Compared to other sequence mod- els, HMMs are appealing since they fully separate the process of generating hidden states from ob- servations, while allowing for exact posterior infer- ence. State-of-the-art systems in NLP have moved away from utilizing latent hidden states and toward deterministic deep neural models. We take several lessons from the success of neural models for NLP tasks: (a) model size is critical for accuracy, e.g. Code available at github.com/harvardnlp/hmm-lm large LSTMs (Zaremba et al., 2014) show marked improvements in performance; (b) the right param- eterization is critically important for representation learning, e.g. a feedforward model (Bengio et al., 2003) can have the same distributional assumptions as an n-gram model while performing significantly better; (c) dropout is key to achieving strong perfor- mance (Zaremba et al., 2014; Merity et al., 2017). We revisit HMMs for language modeling as an alternative to modern neural models, while consid- ering key empirical lessons from these approaches. Towards that goal, we introduce three techniques: a modeling constraint that allows us to use a large number of hidden states while maintaining efficient exact inference, a neural parameterization that im- proves generalization while remaining faithful to the probabilistic structure of the HMM, and a vari- ant of dropout that both improves accuracy and halves the computational overhead during training. Experiments employ HMMs on two language modeling datasets. Our approach allows us to train an HMM with tens of thousands of states while maintaining efficiency and significantly out- performing past HMMs as well as n-gram models. 2 Related Work In order to improve the performance of HMMs on language modeling, several recent papers have combined HMMs with neural networks. Buys et al. …

Content truncated to 500 words