MAISE: A Flexible, Configurable, Extensible Open Source Package for Mass AI System Evaluation

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 130–134, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics MAISE: A Flexible, Configurable, Extensible Open Source Package for Mass AI System Evaluation Omar F. Zaidan Dept. of Computer Science and The Center for Language and Speech Processing Johns Hopkins University Baltimore, MD 21218, USA ozaidan@cs.jhu.edu Abstract The past few years have seen an increasing interest in using Amazon’s Mechanical Turk for purposes of collecting data and perform- ing annotation tasks. One such task is the mass evaluation of system output in a variety of tasks. In this paper, we present MAISE, a package that allows researchers to evalu- ate the output of their AI system(s) using hu- man judgments collected via Amazon’s Me- chanical Turk, greatly streamlining the pro- cess. MAISE is open source, easy to run, and platform-independent. The core of MAISE’s codebase was used for the manual evaluation of WMT10, and the completed package is be- ing used again in the current evaluation for WMT11. In this paper, we describe the main features, functionality, and usage of MAISE, which is now available for download and use. 1 Introduction The ability to evaluate system output is one of the most important aspects of system development. A properly designed evaluation paradigm could help researchers test and illustrate the effectiveness, or lack thereof, of any changes made to their system. The use of an automatic metric, whether it is a sim- ple one such as classification accuracy, or a more task-specific metric such as BLEU and TER for ma- chine translation, has become a standard part of any evaluation of empricial methods. There is also ex- tensive interest in exploring manual evaluation of system outputs, and in making such a process fea- sible and efficient, time- and cost-wise. Such human feedback would also be valuable because it would help identify systematic errors and guide future sys- tem development. Amazon’s Mechanical Turk (MTurk) is a virtual marketplace that allows anyone to create and post tasks to be completed by human workers around the globe. Each instance of those tasks, called a Human Intelligence Task (HIT) in MTurk lingo, typically requires human understanding and perception that machines are yet to achieve, hence making MTurk an example of “artificial artificial intelligence,” as the developers of MTurk aptly put it. Arguably, the most attractive feature of MTurk is the low cost asso- ciated with completing HITs and the speed at which they are completed. Having discovered this venue, many researchers in the fields of artificial intelligence and machine learning see MTurk as a valuable and effective source of annotations, labels, and data, namely the kind requiring human knowledge. One such kind of data is indeed human evalua- tion of system outputs. For instance, if you construct several speech recognition systems, and would like to know how well each of the systems performs, you could create HITs on MTurk that ‘showcase’ the transcriptions obtained by the different systems, and ask annotators to indicate …

Content truncated to 500 words