Active Imitation Learning with Noisy Guidance

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2093–2105 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2093 Active Imitation Learning with Noisy Guidance Kianté Brantley University of Maryland kdbrant@cs.umd.edu Amr Sharaf University of Maryland amr@cs.umd.edu Hal Daumé III University of Maryland Microsoft Research me@hal3.name Abstract Imitation learning algorithms provide state-of- the-art results on many structured prediction tasks by learning near-optimal search policies. Such algorithms assume training-time access to an expert that can provide the optimal ac- tion at any queried state; unfortunately, the number of such queries is often prohibitive, frequently rendering these approaches imprac- tical. To combat this query complexity, we consider an active learning setting in which the learning algorithm has additional access to a much cheaper noisy heuristic that provides noisy guidance. Our algorithm, LEAQI, learns a difference classifier that predicts when the expert is likely to disagree with the heuris- tic, and queries the expert only when neces- sary. We apply LEAQI to three sequence la- beling tasks, demonstrating significantly fewer queries to the expert and comparable (or bet- ter) accuracies over a passive approach. 1 Introduction Structured prediction methods learn models to map inputs to complex outputs with internal dependen- cies, typically requiring a substantial amount of expert-labeled data. To minimize annotation cost, we focus on a setting in which an expert provides labels for pieces of the input, rather than the com- plete input (e.g., labeling at the level of words, not sentences). A natural starting point for this is imita- tion learning-based “learning to search” approaches to structured prediction (Daumé et al., 2009; Ross et al., 2011; Bengio et al., 2015; Leblond et al., 2018). In imitation learning, training proceeds by incrementally producing structured outputs on piece at a time and, at every step, asking the ex- pert “what would you do here?” and learning to mimic that choice. This interactive model comes at a substantial cost: the expert demonstrator must be continuously available and must be able to answer a potentially large number of queries. We reduce this annotation cost by only asking an expert for labels that are truly needed; our al- gorithm, Learning to Query for Imitation (LEAQI, /"li:,tSi:/)1 achieves this by capitalizing on two fac- tors. First, as is typical in active learning (see §2), LEAQI only asks the expert for a label when it is uncertain. Second, LEAQI assumes access to a noisy heuristic labeling function (for instance, a rule-based model, dictionary, or inexpert annota- tor) that can provide low-quality labels. LEAQI operates by always asking this heuristic for a label, and only querying the expert when it thinks the expert is likely to disagree with this label. It trains, simultaneously, a difference classifier (Zhang and Chaudhuri, 2015) that predicts disagreements be- tween the expert and the heuristic (see Figure 1). The challenge in learning the difference classifier is that it must learn based on one-sided feedback: if it predicts that the expert is likely to agree with the heuristic, …

Content truncated to 500 words