How well does active learning actually work? Time-based evaluation of cost-reduction strategies for language documentation.

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 296–305, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP How well does active learning actually work? Time-based evaluation of cost-reduction strategies for language documentation. Jason Baldridge Department of Linguistics The University of Texas at Austin jbaldrid@mail.utexas.edu Alexis Palmer Computational Linguistics Saarland University apalmer@coli.uni-sb.de Abstract Machine involvement has the potential to speed up language documentation. We as- sess this potential with timed annotation experiments that consider annotator exper- tise, example selection methods, and sug- gestions from a machine classifier. We find that better example selection and la- bel suggestions improve efficiency, but ef- fectiveness depends strongly on annota- tor expertise. Our expert performed best with uncertainty selection, but gained lit- tle from suggestions. Our non-expert per- formed best with random selection and suggestions. The results underscore the importance both of measuring annotation cost reductions with respect to time and of the need for cost-sensitive learning meth- ods that adapt to annotators. 1 Introduction Data annotated with linguistically interesting la- bels is used in a wide variety of contexts. Com- putational linguists generally use annotated data as training and evaluation material for natural lan- guage processing systems; corpus linguists use it to test hypotheses about language; documentary linguists create interlinear glossed texts to pre- serve examples of endangered languages and hy- potheses about the grammars of those languages. Regardless of the context, creating annotated data is costly in terms of time and/or money. Since both time and money are undeniably in limited supply, there is a widely shared desire to reduce this cost. Reducing cost involves strategies that do more with fewer human-annotated labels and/or reduce the per-label cost. An example of the former is ac- tive learning, which focuses annotation effort on data points selected by the learner(s) for their ex- pected utility in developing a more accurate model (Settles, 2009). Examples of the latter include providing suggestions from a machine labeler and using extremely cheap human labelers, e.g. with the Amazon Mechanical Turk (Snow et al., 2008). Different techniques may be more or less appli- cable depending on the language being annotated, the kind of labels which are desired (tags, syntac- tic structures, etc.), and the desired use of the an- notated data (e.g., for training models, testing lin- guistic hypotheses, or preserving a language). This paper discusses experiments that measure the effectiveness of machine-aided annotation for language documentation using both active learn- ing simulation experiments and annotation ex- periments which involve actual documentary lin- guists interacting with machine example selec- tion and label suggestion. Specifically, we deal with the task of labeling morphemes of the Mayan language Uspanteko with fine-grained parts-of- speech. We also run active learning simulation experiments for part-of-speech tagging for Dan- ish, Dutch, English, Swedish, and Uspanteko to show the validity of our models and methods in a standard setting. For Uspanteko, we provide re- sults from annotation experiments in which anno- tation cost is measured in terms of the actual an- notation …

Content truncated to 500 words