FIESTA: Fast IdEntification of State-of-The-Art models using adaptive bandit algorithms

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2920–2930 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2920 FIESTA: Fast IdEntification of State-of-The-Art models using adaptive bandit algorithms Henry B. Moss STOR-i Centre for Doctoral Training, Lancaster University Andrew Moore School of Computing and Communications, Lancaster University initial.surname@lancaster.ac.uk David S. Leslie Dept. of Mathematics and Statistics, Lancaster University Paul Rayson School of Computing and Communications, Lancaster University Abstract We present FIESTA, a model selection ap- proach that significantly reduces the compu- tational resources required to reliably identify state-of-the-art performance from large col- lections of candidate models. Despite being known to produce unreliable comparisons, it is still common practice to compare model evaluations based on single choices of random seeds. We show that reliable model selec- tion also requires evaluations based on multi- ple train-test splits (contrary to common prac- tice in many shared tasks). Using bandit the- ory from the statistics literature, we are able to adaptively determine appropriate numbers of data splits and random seeds used to evaluate each model, focusing computational resources on the evaluation of promising models whilst avoiding wasting evaluations on models with lower performance. Furthermore, our user- friendly Python implementation produces con- fidence guarantees of correctly selecting the optimal model. We evaluate our algorithms by selecting between 8 target-dependent sen- timent analysis methods using dramatically fewer model evaluations than current model selection approaches. 1 Introduction and Background Natural Language Processing (NLP) is a field driven by empirical evaluations. Authors are under pressure to demonstrate that their models or meth- ods achieve state-of-the-art performance on a par- ticular task or dataset, which by definition requires reliable model comparison. As models become more numerous, require larger computational re- sources to train, and the performance of compet- ing models gets closer, the task of reliable model selection has not only become more important, but also increasingly difficult. Without full disclosure of model settings and data splits, it is impossible to accurately compare methods and models. To be able to perform meaningful model com- parisons, we need to be able to reliably evaluate models. Unfortunately, evaluating a model is a non-trivial task and the best we can do is to pro- duce noisy estimates of model performance with the following two distinct sources of stochasticity: 1. We only have access to a finite training dataset, however, evaluating a model on its training data leads to severe over-estimates of performance. To evaluate models with- out over-fitting, practitioners typically ran- domly partitioning data into independent training and testing sets, producing estimates that are random quantities with often high variability for NLP problems (Moss et al., 2018). Although methods like bootstrapping (Efron and Tibshirani, 1994) and leave-one- out cross validation (Kohavi, 1995) can pro- vide deterministic estimates of performance, they require the fitting of a large number of models and so are not computationally fea- sible for the complex models and large data prevalent in NLP. Standard NLP model eval- uation strategies …

Content truncated to 500 words