Optimal Data Set Selection: An Application to Grapheme-to-Phoneme Conversion

Proceedings of NAACL-HLT 2013, pages 1196–1205, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Optimal Data Set Selection: An Application to Grapheme-to-Phoneme Conversion Young-Bum Kim and Benjamin Snyder University of Wisconsin-Madison {ybkim,bsnyder}@cs.wisc.edu Abstract In this paper we introduce the task of unla- beled, optimal, data set selection. Given a large pool of unlabeled examples, our goal is to select a small subset to label, which will yield a high performance supervised model over the entire data set. Our first proposed method, based on the rank-revealing QR ma- trix factorization, selects a subset of words which span the entire word-space effectively. For our second method, we develop the con- cept of feature coverage which we optimize with a greedy algorithm. We apply these methods to the task of grapheme-to-phoneme prediction. Experiments over a data-set of 8 languages show that in all scenarios, our selec- tion methods are effective at yielding a small, but optimal set of labelled examples. When fed into a state-of-the-art supervised model for grapheme-to-phoneme prediction, our meth- ods yield average error reductions of 20% over randomly selected examples. 1 Introduction Over the last 15 years, supervised statistical learning has become the dominant paradigm for building nat- ural language technologies. While the accuracy of supervised models can be high, expertly annotated data sets exist for a small fraction of possible tasks, genres, and languages. The would-be tool builder is thus often faced with the prospect of annotating data, using crowd-sourcing or domain experts. With limited time and budget, the amount of data to be an- notated might be small, especially in the prototyping stage, when the exact specification of the prediction task may still be in flux, and rapid prototypes are desired. In this paper, we propose the problem of unsuper- vised, optimal data set selection. Formally, given a large set X of n unlabeled examples, we must select a subset S ⊂ X of size k ≪ n to label. Our goal is to select such a subset which, when labeled, will yield a high performance supervised model over the entire data set X. This task can be thought of as a zero-stage version of active learn- ing: we must choose a single batch of examples to label, without the benefit of any prior labelled data points. This problem definition avoids the practical complexity of the active learning set-up (many it- erations of learning and labeling), and ensures that the labeled examples are not tied to one particular model class or task, a well-known danger of active learning (Settles, 2010). Alternatively, our methods may be used to create the initial seed set for the ac- tive learner. Our initial testbed for optimal data set selec- tion is the task of grapheme-to-phoneme conver- sion. In this task, we are given an out-of-vocabulary word, with the goal of predicting a sequence of phonemes corresponding to its pronunciation. As training data, we are given a pronunciation dic- tionary listing words alongside corresponding se- quences of phones, representing canonical pronun- …

Content truncated to 500 words