Reducing Confusion in Active Learning for Part-Of-Speech Tagging

Reducing Confusion in Active Learning for Part-Of-Speech Tagging Aditi Chaudhary1, Antonios Anastasopoulos2,∗, Zaid Sheikh1, Graham Neubig1 1Language Technologies Institute, Carnegie Mellon University 2Department of Computer Science, George Mason University {aschaudh,zsheikh,gneubig}@cs.cmu.edu antonis@gmu.edu Abstract Active learning (AL) uses a data selection algorithm to select useful training samples to minimize annotation cost. This is now an es- sential tool for building low-resource syntactic analyzers such as part-of-speech (POS) tag- gers. Existing AL heuristics are generally de- signed on the principle of selecting uncertain yet representative training instances, where annotating these instances may reduce a large number of errors. However, in an empirical study across six typologically diverse lan- guages (German, Swedish, Galician, North Sami, Persian, and Ukrainian), we found the surprising result that even in an oracle scenario where we know the true uncertainty of predic- tions, these current heuristics are far from optimal. Based on this analysis, we pose the problem of AL as selecting instances that maxi- mally reduce the confusion between particular pairs of output tags. Extensive experimenta- tion on the aforementioned languages shows that our proposed AL strategy outperforms other AL strategies by a significant margin. We also present auxiliary results demonstrat- ing the importance of proper calibration of models, which we ensure through cross-view training, and analysis demonstrating how our proposed strategy selects examples that more closely follow the oracle data distribution. The code is publicly released here.1 1 Introduction Part-of-speech (POS) tagging is a crucial step for language understanding, both being used in auto- matic language understanding applications such as named entity recognition (NER; Ankita and Nazeer, 2018) and question answering (QA; Wang et al., 2018), but also being used in manual lan- ∗Work done at Carnegie Mellon University. 1https://github.com/Aditi138/CRAL. guage understanding by linguists who are attempt- ing to answer linguistic questions or document less-resourced languages (Anastasopoulos et al., 2018). Much prior work (Huang et al., 2015; Bohnet et al., 2018) on developing high-quality POS taggers uses neural network methods, which rely on the availability of large amounts of labelled data. However, such resources are not readily available for the majority of the world’s 7000 languages (Hammarstr¨om et al., 2018). Fur- thermore, manually annotating large amounts of text with trained experts is an expensive and time-consuming task, even more so when lin- guists/annotators might not be native speakers of the language. Active Learning (Lewis, 1995; Settles, 2009, AL) is a family of methods that aim to train effec- tive models with less human effort and cost by selecting such a subset of data that maximizes the end model performance. Although many methods have been proposed for AL in sequence labeling (Settles and Craven, 2008; Marcheggiani and Arti`eres, 2014; Fang and Cohn, 2017), through an empirical study across six typologically di- verse languages we show that within the same task setup these methods perform inconsistently. Furthermore, even in an oracle scenario, where we have access to the true labels during data selection, existing methods are far from optimal. We posit that the primary reason for this inconsistent performance …

Content truncated to 500 words