Learning how to Active Learn: A Deep Reinforcement Learning Approach

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 595–605 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Learning how to Active Learn: A Deep Reinforcement Learning Approach Meng Fang and Yuan Li and Trevor Cohn School of Computing and Information Systems The University of Melbourne meng.fang@unimelb.edu.au, yuanl4@student.unimelb.edu.au, t.cohn@unimelb.edu.au Abstract Active learning aims to select a small sub- set of data for annotation such that a classi- fier learned on the data is highly accurate. This is usually done using heuristic selec- tion methods, however the effectiveness of such methods is limited and moreover, the performance of heuristics varies between datasets. To address these shortcomings, we introduce a novel formulation by re- framing the active learning as a rein- forcement learning problem and explicitly learning a data selection policy, where the policy takes the role of the active learning heuristic. Importantly, our method allows the selection policy learned using simu- lation on one language to be transferred to other languages. We demonstrate our method using cross-lingual named entity recognition, observing uniform improve- ments over traditional active learning. 1 Introduction For most Natural Language Processing (NLP) tasks, obtaining sufficient annotated text for train- ing accurate models is a critical bottleneck. Thus active learning has been applied to NLP tasks to minimise the expense of annotating data (Thomp- son et al., 1999; Tong and Koller, 2001; Settles and Craven, 2008). Active learning aims to reduce cost by identifying a subset of unlabelled data for anno- tation, which is selected to maximise the accuracy of a supervised model trained on the data (Settles, 2010). There have been many successful appli- cations to NLP, e.g., Tomanek et al. (2007) used an active learning algorithm for CoNLL corpus to get an F1 score 84% with a reduction of annotation cost of about 48%. In prior work most active learn- ing algorithms are designed for English based on heuristics, such as using uncertainty or informa- tiveness. There has been comparatively little work done about how to learn the active learning strat- egy itself. It is no doubt that active learning is extremely important for other languages, particularly low- resource languages, where annotation is typically difficult to obtain, and annotation budgets more modest (Garrette and Baldridge, 2013). Such set- tings are a natural application for active learning, however there is little work to this end. A poten- tial reason is that most active learning algorithms require a substantial ‘seed set’ of data for learning a basic classifier, which can then be used for ac- tive data selection. However, given the dearth of data in the low-resource setting, this assumption can make standard approaches infeasible. In this paper,1 we propose PAL, short for Pol- icy based Active Learning, a novel approach for learning a dynamic active learning strategy from data. This allows for the strategy to be applied in other data settings, such as cross-lingual applica- tions. Our algorithm does not use a fixed heuris- tic, but instead learns how to actively …

Content truncated to 500 words