Efficient Hyper-parameter Optimization for NLP Applications

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2112–2117, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Efficient Hyper-parameter Optimization for NLP Applications Lidan Wang1, Minwei Feng1, Bowen Zhou1, Bing Xiang1, Sridhar Mahadevan2,1 1IBM Watson, T. J. Watson Research Center, NY 2College of Information and Computer Sciences, U. of Massachusetts Amherst, MA {wangli,mfeng,zhou,bingxia}@us.ibm.com mahadeva@cs.umass.edu Abstract Hyper-parameter optimization is an im- portant problem in natural language pro- cessing (NLP) and machine learning. Re- cently, a group of studies has focused on using sequential Bayesian Optimization to solve this problem, which aims to reduce the number of iterations and trials required during the optimization process. In this paper, we explore this problem from a dif- ferent angle, and propose a multi-stage hyper-parameter optimization that breaks the problem into multiple stages with in- creasingly amounts of data. Early stage provides fast estimates of good candidates which are used to initialize later stages for better performance and speed. We demon- strate the utility of this new algorithm by evaluating its speed and accuracy against state-of-the-art Bayesian Optimization al- gorithms on classification and prediction tasks. 1 Introduction Hyper-parameter optimization has been receiv- ing an increasingly amount of attention in the NLP and machine learning communities (Thorn- ton et al., 2013; Komer et al., 2014; Bergstra et al., 2011; Bardenet et al., 2013; Zheng et al., 2013). The performance of learning algorithms depend on the correct instantiations of their hyper- parameters, ranging from algorithms such as lo- gistic regression and support vector machines, to more complex model families such as boosted regression trees and neural networks. While hyper-parameter settings often make the differ- ence between mediocre and state-of-the-art per- formance (Hutter et al., 2014), it is typically very time-consuming to find an optimal setting due to the complexity of model classes, and the amount of training data available for tuning. The issue is particularly important in large-scale problems where the size of the data can be so large that even a quadratic running time is prohibitively large. Recently several sequential Bayesian Op- timization methods have been proposed for hyper-parameter search (Snoek et al., 2012; Eggensperger et al., 2015; Brochu et al., 2010; Hutter et al., 2011; Eggensperger et al., 2014). The common theme is to perform a set of it- erative hyper-parameter optimizations, where in each round, these methods fit a hyper-parameter response surface using a probabilistic regression function such as Gaussian Process (Snoek et al., 2012) or tree-based models (Hutter et al., 2011), where the response surface maps each hyper- parameter setting to an approximated accuracy. The learned regression model is then used as a cheap surrogate of the response surface to quickly explore the search space and identify promising hyper-parameter candidates to evaluate next in or- der to enhance validation accuracy. While these methods have enjoyed great success compared to conventional random search (Bergstra et al., 2012; Bengio et al., 2013) and grid search algorithms by significantly reduc- ing the number of iterations and trials required during …

Content truncated to 500 words