Bayesian Optimization of Text Representations

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2100–2105, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Bayesian Optimization of Text Representations Dani Yogatama Lingpeng Kong School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213, USA {dyogatama,lingpenk}@cs.cmu.edu Noah A. Smith Computer Science & Engineering University of Washington Seattle, WA 98195, USA nasmith@cs.washington.edu Abstract When applying machine learning to prob- lems in NLP, there are many choices to make about how to represent input texts. They can have a big effect on perfor- mance, but they are often uninteresting to researchers or practitioners who simply need a module that performs well. We ap- ply sequential model-based optimization over this space of choices and show that it makes standard linear models competitive with more sophisticated, expensive state-of- the-art methods based on latent variables or neural networks on various topic classifica- tion and sentiment analysis problems. Our approach is a first step towards black-box NLP systems that work with raw text and do not require manual tuning. 1 Introduction NLP researchers and practitioners spend a consid- erable amount of time comparing machine-learned models of text that differ in relatively uninteresting ways. For example, in categorizing texts, should the “bag of words” include bigrams, and is tf-idf weighting a good idea? In learning word embed- dings, distributional similarity approaches have been shown to perform competitively with neural network models when the hyperparameters (e.g., context window, subsampling rate, smoothing con- stant) are carefully tuned (Levy et al., 2015). These choices matter experimentally, often leading to big differences in performance, with little consistency across tasks and datasets in which combination of choices works best. Unfortunately, these differ- ences tell us little about language or the problems that machine learners are supposed to solve. We propose that these decisions can be auto- mated in a similar way to hyperparameter selection (e.g., choosing the strength of a ridge or lasso regu- larizer). Given a particular text dataset and classi- fication task, we show a technique for optimizing over the space of representational choices, along with other “nuisances” that interact with these de- cisions, like hyperparameter selection. For exam- ple, using higher-order n-grams means more fea- tures and a need for stronger regularization and more training iterations. Generally, these decisions about instance representation are made by humans, heuristically; our work seeks to automate them, not unlike Daelemans et al. (2003), who proposed to use genetic algorithms to optimize representational choices. Our technique instantiates sequential model- based optimization (SMBO; Hutter et al., 2011). SMBO and other Bayesian optimization ap- proaches have been shown to work well for hyper- parameter tuning (Bergstra et al., 2011; Hoffman et al., 2011; Snoek et al., 2012). Though popular in computer vision (Bergstra et al., 2013), these techniques have received little attention in NLP. We apply it to logistic regression on a range of topic and sentiment classification tasks. Consis- tently, our method finds representational choices that perform better than linear baselines previously reported in the literature, …

Content truncated to 500 words