Segmentation for Efficient Supervised Language Annotation with an Explicit Cost-Utility Tradeoff

Segmentation for Efficient Supervised Language Annotation with an Explicit Cost-Utility Tradeoff Matthias Sperber1, Mirjam Simantzik2, Graham Neubig3, Satoshi Nakamura3, Alex Waibel1 1Karlsruhe Institute of Technology, Institute for Anthropomatics, Germany 2Mobile Technologies GmbH, Germany 3Nara Institute of Science and Technology, AHC Laboratory, Japan matthias.sperber@kit.edu, mirjam.simantzik@jibbigo.com, neubig@is.naist.jp s-nakamura@is.naist.jp, waibel@kit.edu Abstract In this paper, we study the problem of manu- ally correcting automatic annotations of natu- ral language in as efficient a manner as pos- sible. We introduce a method for automati- cally segmenting a corpus into chunks such that many uncertain labels are grouped into the same chunk, while human supervision can be omitted altogether for other segments. A tradeoff must be found for segment sizes. Choosing short segments allows us to reduce the number of highly confident labels that are supervised by the annotator, which is useful because these labels are often already correct and supervising correct labels is a waste of effort. In contrast, long segments reduce the cognitive effort due to context switches. Our method helps find the segmentation that opti- mizes supervision efficiency by defining user models to predict the cost and utility of su- pervising each segment and solving a con- strained optimization problem balancing these contradictory objectives. A user study demon- strates noticeable gains over pre-segmented, confidence-ordered baselines on two natural language processing tasks: speech transcrip- tion and word segmentation. 1 Introduction Many natural language processing (NLP) tasks re- quire human supervision to be useful in practice, be it to collect suitable training material or to meet some desired output quality. Given the high cost of human intervention, how to minimize the supervi- sion effort is an important research problem. Previ- ous works in areas such as active learning, post edit- (a) It was a bright cold (they) in (apron), and (a) clocks were striking thirteen. (b) It was a bright cold (they) in (apron), and (a) clocks were striking thirteen. (c) It was a bright cold (they) in (apron), and (a) clocks were striking thirteen. Figure 1: Three automatic transcripts of the sentence “It was a bright cold day in April, and the clocks were strik- ing thirteen”, with recognition errors in parentheses. The underlined parts are to be corrected by a human for (a) sentences, (b) words, or (c) the proposed segmentation. ing, and interactive pattern recognition have inves- tigated this question with notable success (Settles, 2008; Specia, 2011; Gonz´alez-Rubio et al., 2010). The most common framework for efficient anno- tation in the NLP context consists of training an NLP system on a small amount of baseline data, and then running the system on unannotated data to estimate confidence scores of the system’s predictions (Set- tles, 2008). Sentences with the lowest confidence are then used as the data to be annotated (Figure 1 (a)). However, it has been noted that when the NLP system in question already has relatively high accu- racy, annotating entire sentences can be wasteful, as most words will already be correct (Tomanek and Hahn, 2009; Neubig et al., 2011). In …

Content truncated to 500 words