Overcoming the Memory Bottleneck in Distributed Training of Latent Variable Models of Text

Proceedings of NAACL-HLT 2013, pages 579–584, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Overcoming the Memory Bottleneck in Distributed Training of Latent Variable Models of Text Yi Yang Northwestern University Evanston, IL yiyang@eecs.northwestern.edu Alexander Yates Temple University Philadelphia, PA yates@temple.edu Doug Downey Northwestern University Evanston, IL ddowney@eecs.northwestern.edu Abstract Large unsupervised latent variable models (LVMs) of text, such as Latent Dirichlet Al- location models or Hidden Markov Models (HMMs), are constructed using parallel train- ing algorithms on computational clusters. The memory required to hold LVM parameters forms a bottleneck in training more powerful models. In this paper, we show how the mem- ory required for parallel LVM training can be reduced by partitioning the training corpus to minimize the number of unique words on any computational node. We present a greedy document partitioning technique for the task. For large corpora, our approach reduces mem- ory consumption by over 50%, and trains the same models up to three times faster, when compared with existing approaches for paral- lel LVM training. 1 Introduction Unsupervised latent variable models (LVMs) of text are utilized extensively in natural language process- ing (Griffiths and Steyvers, 2004; Ritter et al., 2010; Downey et al., 2007; Huang and Yates, 2009; Li and McCallum, 2005). LVM techniques include Latent Dirichlet Allocation (LDA) (Blei et al., 2003), Hid- den Markov Models (HMMs) (Rabiner, 1989), and Probabilistic Latent Semantic Analysis (Hofmann, 1999), among others. LVMs become more predictive as they are trained on more text. However, training LVMs on mas- sive corpora introduces computational challenges, in terms of both time and space complexity. The time complexity of LVM training has been addressed through parallel training algorithms (Wolfe et al., 2008; Chu et al., 2006; Das et al., 2007; Newman et al., 2009; Ahmed et al., 2012; Asuncion et al., 2011), which reduce LVM training time through the use of large computational clusters. However, the memory cost for training LVMs re- mains a bottleneck. While LVM training makes se- quential scans of the corpus (which can be stored on disk), it requires consistent random access to model parameters. Thus, the model parameters must be stored in memory on each node. Because LVMs in- clude a multinomial distribution over words for each latent variable value, the model parameter space in- creases with the number of latent variable values times the vocabulary size. For large models (i.e., with many latent variable values) and large cor- pora (with large vocabularies), the memory required for training can exceed the limits of the commod- ity servers comprising modern computational clus- ters. Because model accuracy tends to increase with both corpus size and model size (Ahuja and Downey, 2010; Huang and Yates, 2010), training accurate lan- guage models requires that we overcome the mem- ory bottleneck. We present a simple technique for mitigating the memory bottleneck in parallel LVM training. Ex- isting parallelization schemes begin by partitioning the training corpus arbitrarily across computational nodes. In this paper, we show how to reduce mem- ory footprint by …

Content truncated to 500 words