Efficient Benchmarking of NLP APIs using Multi-armed Bandits

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 408–416, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Efficient Benchmarking of NLP APIs using Multi-armed Bandits Gholamreza Haffari and Tuan Tran and Mark Carman Faculty of Information Technology, Monash University Clayton, VICTORIA 3800, Australia {gholamreza.haffari, mark.carman}@monash.edu tdtra18@student.monash.edu Abstract Comparing NLP systems to select the best one for a task of interest, such as named entity recognition, is critical for practition- ers and researchers. A rigorous approach involves setting up a hypothesis testing scenario using the performance of the sys- tems on query documents. However, often the hypothesis testing approach needs to send a large number of document queries to the systems, which can be problematic. In this paper, we present an effective al- ternative based on the multi-armed ban- dit (MAB). We propose a hierarchical gen- erative model to represent the uncertainty in the performance measures of the com- peting systems, to be used by Thompson Sampling to solve the resulting MAB. Ex- perimental results on both synthetic and real data show that our approach requires significantly fewer queries compared to the standard benchmarking technique to identify the best system according to F- measure. 1 Introduction F-measureAs new NLP systems are (continually) introduced for a task of interest, such as named entity recognition (NER), it is crucial for prac- tioneers and researchers to select the best system. These systems may be designed based on different models and/or learning algorithms. For instance, due to recent advancement in NER research, sev- eral NER systems have been proposed and then supported in APIs such as OpenNLP (Ingersoll et al., 2013), Stanford NER (Finkel et al., 2005), ANNIE (Cunningham et al., 2002) and Meaning Cloud (MeaningCloud-LLC, 1998) to name a few. Often, the competing NLP systems are bench- marked according to their average performance measure, e.g. F-measure capturing both Precision and Recall, across a set of example documents. Each document produces a single F-measure and the true performance of the system is considered to be the expected value across all possible doc- uments from the domain. Performance on indi- vidual documents correspond to samples from the performance distribution of the system, and can then be used to determine the best system (or set of systems should the highest performing system not be unique) using rigorous hypothesis testing. However, this approach usually requires querying each competing system with a large number of documents, which can be problematic if either the number of test documents is limited, or the sys- tems are implemented as APIs by a third party and performing each query incurs a cost. In this paper, we present a statistically effective method to identify the best system from a pool of systems. Our approach requires significantly fewer example documents to reach similar guarantees as the traditional hypothesis testing set up, hence re- ducing the cost and increasing the speed of infer- ence. Inspired by the previous work (Scott, 2015; Gabillon …

Content truncated to 500 words